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ABSTRACT 

The primary objective of this project was to design a 
system to provide information services that would fulfill the 
information requirements of federal planners, instructional material 
developers, training program directors and instructors, students, 
interns, employers, and practitioners concerned with problems 
relating to personnel and training for educational research, 
development, diffusion and evaluation (RDDSE) • Chapter I of the 
report outlines the project and summarizes previous reports on the 
planning and preliminary design phases. Chapter II provides a 
description of the target audiences and an analysis of their 
requirements. Chapter III describes the piloi, field test of 1^ 
dissemination alternatives; presents the results; and makes 
recommendations, with cost estimates and rationale, for each 
promising alternative. Chapter IV provides insight into the 
perceptions of representatives from each of the five key target 
groups about priorities for serving each of 16 subclasses of 
potential information users. Chapter V analyzes the amount of 
information and instructional resources existing in several RDDSE 
content categories. Chapter VI deals with selecting the best and most 
useful items for acquisition and dissemination. The last chapter 
summarizes the data qraphically and presents the conclusions of the 
study. (Related documents are SP 007 655, 007 656, 007 658, and 007 
659.) (DDO) 
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EXECUTIVE SUMMARY 



How can the National Institute of Education (NIE) provide or enhance 
access to personnel and training information and instructional resources 
that would improve the quality, competence, and utilization of the per- 
sonnel needed for educational research, development, diffusion, and 
evaluation (RDD&E)? This basic question, raised by the NIE Task Force on 
Researcher Training, resulted 'In the commissioning of this project. 
Initially, NIE viewed the project as an inquiry into the feasibility of 
establishing an information and instructional materials center for edu- 
cational RDD&E personnel and training. Such a center might serve a variety 
of target audiences, including sponsors, R&D training performers, trainers, 
students, employers, and educational practitioners. As the project pro- 
gressed, however, it was determined, not only as a result of information 
developed by the project, but also in response to NIE priorities, that 
immediate establishment of an information center would not be required. 
Resources which had been reserved for advanced design work on the ca^Jgr 
and for preparation of its implementation guidelines were then redirected 
to several other specific problems. Although the project focus remained 
on "personnel and training," our findings have broader implications for 
the problem of R&D communication and technology transfer within the entire 
field of educational RDDSE. 

The project was a very modest one, involving less than one-and-a-half 
men years of professional effort. The field test samples are small, but 
representative of the selected target audiences. Because selected alter- 
native information services and prototype information products were actually 
provided to test users, who were interviewed individually after using or 
examining them, the reported user reactions are to tangibles rather than 
abstractions- The field test results are also set in the context of a much 
broader analysis of what is known or can be inferred about user require- 
ments, needs for information, estimated quantities of available literature 
and instructional materials in different RDD&E content areas, etc. 

Chapter I outlines the project and summarizes previous reports on the 
planning and the prel Iminary design phases. Chapter II provides a descrip- 
tion of the target audiences and an analysis of their requirements. Chapter 
III describes the pilot field test of 14 dissemination alternatives, presents 
the results, and makes recommendations, with cost estimates and rationale, 
for each promising alternative. Chapter IV provides insight into the per- 
c^.ptions of representatives from the five key target groups about priorities 
for serving each of 16 subclasses of potential information users. 

Chapter V turns from the examination of users and their requirements 
to the analysis of how much information and instructional resources exist 
in several RDD&E content categories. Relative and absolute estimates of 
quantities are presented. Chapter VI deals with a derivative problem: 
how can one select the "best'* or, more importantly, the most "useful" set 
of items for acquisition or dissemination? This chapter, which focuses 
especially on the evaluation of instructional materials, also contains an 
analysis of the decision problem and provides general recommendations for 
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selection and evaluation. Chapter VII graphically summarizes the data 
presented in previous chapters. In addition, it presents a brief con- 
clusion which attempts to relate the study to a more general set of 
problems regarding NIE options for fostering communication within the 
"problem-oriented" R&D network. 
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I. INTRODUCTION 
A. Objectives 

The primary objective of this project was to design a system to 
provide information services that would fulfill the information requirements 
of federal planners, instructional materials developers, training program 
directors and instructors, students, interns, employers, and practitioners 
concerned with problems relating to personnel and training for educational 
research, development, diffusion, and evaluation (RDD&E). 

Initially the National Institute of Education (NIE) viewed the project 
as an inquiry into the feasibility of starting an information center for 
educational RDD&E personnel and training information and instructional 
resources. Four phases were projected: (1) a planning phase (July 1- 
August 31, 1972), (2) a preliminary design phase (September 1 -November 30, 
1972), (3) an advance design phase (December 1, 1972-January 31, 1973), and 
(4) a final report phase (February l-February 28, 1S73). The eventual 
establishment of such an information center was a conceivable outcome of the 
project. 

During the progress of the project, partly as a result of information 
developed by the project, but also in response to NIE reconmendations and 
federal policy toward support of training, we recognized that the ijTWediate 
establishment of an information center would not be required. Consequently 
the experimental pilot tests, initiated during the preliminary design phase, 
were extended into December 1972 and January 1973, and the resources allocated 
for the last two phases were redirected toward providing NIE Researcher 
Training Task Force staff with information bearing more directly on instructional 
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resources. The information included recomnendations for evaluation and estimates 
of quantities of instructional materials in different content categories. 
Specific recommendations for inexpensive dissemination alternatives were also 
made. Finally, recommendations that might help NIE to establish a "core" 
RDD&E information collection were also submitted. 

B. List of Accomplishments 
The project has achieved the following: 

1. Assessed user needs, including sponsors, researchers, developers, 
and trainers, and provided priority listings of user groups, and 
information content categories. 

2. Created an experimental collection of over 1,000 items of literature 
and other materials representative of the types of information to be 
handled by the proposed system. 

3. Selected, indexed, abstracted, and classified over 200 items to be 
used in pilot testing dissemination alternatives. 

4. Examined alternatives for acquiring, evaluating, indexing, storing 
and retrieving information and materials. 

5. Examined alternative procedures for disseminating information to 
various types of users. 

6. Experimentally pilot tested a total of fourteen different activities 
and media for information dissemination. 

7. Through a series of meetings and progress reports, involved National 
Institute of Education (NIE) staff, project technical consultants, 
and representatives of user groups in selection and evaluation of 
design alternatives. 

8. Prepared a set of reconwended promising alternatives for dissemination 
and developed a justification and a cost estimate for each alternative. 
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9. Investigated in depth the problems of evaluating instructional 
materials. 

10. Prepared recommended lists of periodicals, references, and books for 
the craation of a "core" collection in educational RDD&E training. 

11. Submitted five interim reports, this final report, and copies of all 
test products. 

In the following sections, the activities and reports of the project are 
summarized. 

C. Summary of Planning Phase Report 

Introduction 

The specific intent of the Planning Phase Report was to provide a basis 
for continuing dialogue between project staff and NIE sponsors, as well as 
representatives of user groups. 

Since policy problems encountered during the planning effort impinged on 
larger issues of development and utilization of RDD&E talent, scientific and 
technical cownuni cation, dissemination and utilization of research and develop- 
ment (R&D) products, allocation of R&D respurces, and economic justification 
for R&D investment, this report also provides useful perspectives on problems 
faced by nearly all of the NIE task forces. 
Summary of the Repo rt 

The report consists of four main chapters and supportive appendices. 
1 . Chapter I: Introduction and Overview 

This chapter lists project objectives; outlines the phases of the project; 
provides background information on the Research Training Program; sunwarizes 
the information needs of, and benefits for, the target audiences who might be 
served by a personnel and training information system; outlines the design 
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requirements and approach; and discusses the relevance of the project to HIE plans 
and programs. 

2. Chapter II: Definition of the Problem 

Chapter II may hold the greatest interest for NIE planners. There is an 
initial discussion of the "problem space" in terms of ten systems which affect 
the transfer of scientific knowledge. The concepts of horizontal and vertical 
flow of knowledge are introduced within the contexts of the educational R&D, 
the personnel and training, and the practice improvement networks. 

A second section provides perspective on the context for, and emergent 
goals of, the Research Training Program, as well as on NIE's dissemination and 
utilization program (including ERIC). 

A third section reviews the resources and constraints which may be signifi- 
cant. It includes discussions of legal, NIE policy, other federal policy, 
financial, copyright, forms clearance, printing and binding, and other contract 
constraints. The potential resources discussed are nonfiscal, including the 
cultural system, political systems, membership groups and reference groups, 
invisible colleges, formal organizations, formal information systems, and 
federal and NIE policy. 

The chapter concludes with an examination, and request for careful reader 
review, of the following proposed design concepts, which may markedly bias the 
design effort: commitment to a systems approach, user orientation, compatibility 
with other NIE information systems, economy, functional emphasis, design for 
flexibility and improvement, and the need for experimentation and evaluation. 
The anticipated outcome is an information system which will have these charac- 
teristics: 
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""itwill actively acquire important personnel and training data and 
information; 

""it will select, evaluate, synthesize, and reformulate personnel and 
training information, tailoring it to the requirements of priority 
target groups to facilitate the vertical knowledge flow; 

""it will actively disseminate Information to those who may need it; 

""it will arrange for, and possibly support awareness of and access 
to, validated methodology, models, and instructional meterials; and 

^it will stimulate the formation and support the maintenance of 
invisible colleges among members of key target groups. 

3. Chapter III: Analysis of User Requirements 

The users of the proposed information system are identified as a compos- 
ite of six different groups of RDD&E personnel types: (1) sponsors, 
(2) performers, (3) trainers, (4) students/interns, (5) employers, and (6) 
practitioners. A description and an estimate of the size of each group, as 
well as its subgroups, are included. A subsequent section reviews each group's 
personnel and training information needs. A discussion follows on the use of 
information sources and the problems users may encounter in searching sources 
for relevant information on educational RDD&E personnel and training. This 
chapter concludes with a note on the distinction between information need 
and information demand which, in turn, points to the need not only for policy 
analysis, but also for studies of user's articulated demands or desires for 
information. 

4* Chapter IV: Preliminary Analysis of Alternatives 

This chapter introduces the information systems concepts of Information 
(content) flow, functions, and nodes (originators, processors, and users). 

The concept of Information flow is analyzed first, in terms of Information 
structurei level of formalization, and content. The Implications of these 
components for analysis and classification of the information flow of educa- 
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tional RDD&E content in general, and personnel and training content in 
particular, are examined. 

The explanation of information system functions consists of brief 
descriptions of the following: acquisition; screening and evaluation; 
surrogation; announcement and dissemination; index operations; document 
and materials management and delivery; reference and special services; 
policy formation, management, and administration; system evaluation; 
modi ri cation and improvement; and resource procurement. 

The concept of information system nodes and the three roles of orig- 
inator, processor, and user are then briefly introduced. 

This presentation of the conceptual framework is succeeded by 4 dis- 
cussion of the problem of information synthesis. Two important distinctions 

between the library and the information center are made: (a) the former's 
collection is a function of its knowledge base while the letter's collection 
is a function of current user needs, and (b) the library's function is to 
archive information while the center's is to disseminate it. Following the pre- 
sentation of a flow chart of information system functions, a basis for 
comparison of information system models is provided. Three system dimen- 
sions — orientation toward the source of the knowledge base, degree of 
centralization of resources, and emphasis on formal vs. informal sources 
of information — are discussed. A tentative set of user-oriented activities 
(as opposed to operator-oriented functions) is also introduced. 

A device for generating a very large population of system objectives 
is then proposed. It is based on the concepts of activities , content of 
information flow, and user groups . Methods for coping with the more than 
four billion objectives which might be generated are discussed. 
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The chapter concludes with a discussion of model evaluation- A number of 
related NIE policy questions, which will require answers, are identified. 
The importance of criteria and measures is also briefly discussed. A symbolic 
model is presented to specify the major elements which enter into the evalua- 
tion of system components and overall information systems models. 
5. Appendices 

Five appendices contain reports of project progress, descriptions of the 
filing system and the experimental collection, and further analyses of 
alternatives for acquisition, indexing, storage, retrieval, evaluation, and 
dissemination. 

D. Summary of Preliminary Design Phase Report 

Introduction 

The intent of this report, like that of the previous one, is to provide 
a basis for continuing dialogue between project staff and NIE sponsors, as 
well as representative users. This report focuses on the selection of 
dissemination alternatives and the development of a plan for pilot testing 
them. 

Summary of the Report 

1 . Chapter I: Introduction and Review 

Chapter I reviews the objectives of the project, summarizes the previous 
report on the planning phase, and presents responses of the NIE staff to our 
questions regarding their judgment of priorities for groups and activities 
including: (1) target groups, (2) target subgroups, (3) alternative activities 
for the information system, and (4) categories of information content in terms 
of information needs of NIE/USOE sponsors. The chapter concludes with the 
contents of a memorandum prepared by NIE/TFRT, responding to policy questions 
in the Planning Report. 
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2. Chapter II: The Preliminary Design Conference 

This chapter narrates the transactions and recomnendations of the 
Preliminary Design Conference. After studying the Planning Report and 
examining the responses by NIE staff, the conference participants examined 
the array of alternative products and services that had been developed 
during the previous project phase. Two advocate design teams then prepared 
independent analyses and recommendations for functions, services, and 
products of an information center. These recommendations led to the choice 
of pilot test dissemination alternatives. 

3. Chapter III: Plan for Pilot Test of Dissemination Alternatives 

The third chapter describes the plan for a pilot test of dissemination 
alternatives. Proposed products and services are listed below In four groups 

a. Announcement and Current Awareness 

"^Personnel and Training Newsletter 
"^Personnel and Training Abstracts 

b. Services 

®Hot-line Query Service 
®Mail Query Service 
®Walk-in Query Service 
Dissemination and Utilization Service 

c. Informal Coirmuni cation 

''Conferences and meetings 
''Teleconwunl cation 

d. Products 

^Instructional Materials Catalog 
^ Grant Package 
""Mini-Collection 
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"^Case Study Source Book 
""Li terature Source Book 
"^Di rectory of Training Opportunities 
Each dissemination alternative is briefly described in terms of 
purpose, rationale, content, and pilot-test format. 

The following pilot-test samples and the rationale for selecting them 
are then described: (1) NIE/USOE sponsors, (2) NIE/TFRT sponsored projects, 
(3) degree training programs, and (4) nondegree training programs. The 
chapter concludes with a schedule and a description of the plan for analysis. 
4. Appendices 

Four appendices contain reports of project progress, response forms and 
data on NIE/USOE staff responses about target groups, activity and information 
content priorities. Preliminary Design Conference notes, and descriptions and 
illustrations of the selection of pilot dissemination products and services. 

E» Advanced Design Phase Reports 

By December 1972, it had become apparent at NIE that new initiatives 
in support of RDD&E training would be made only after thorough review and 
evaluation of current efforts and only if strong justification for needs and 
the cost effectiveness of proposed solutions could be submitted. Moreover, 
preliminary results of the pilot test of the dissemination alternatives 
raised serious questions regarding the need for a full-scale information center 
as opposed to more modest alternatives. 

By mutual agreement, the original project schedule was modified. It had 
anticipated development of a plan for an information center, together with 
the delivery of a handbook of detailed procedures for implementing and main- 
taining the center. Instead, the pilot test was extended through what had 
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been the time period for the advanced design phase (December 1972 - January 
1.973) and the resources allocated for this phase were redirected toward these 
activities: 

1. Completion and reporting of the user study of dissemination 
alternatives, 

2. Preparation of recommendations, together with cost estimates 
for promising dissemination alternatives, 

3. Analysis of the content and quantity of RDD&E training resources, 

4. Examination of problems and issues, and preparation of recommenda- 
tions for the evaluation of RDD&E instructional materials, 

5. Preparation of lists of recommended journals, reference works, and 
books needed to establish a "core" educational RDD&E training 
information collection, 

6. Completion of other project activities and preparation of a final 
report. 

Interim reports on activities 1-5 above were submitted in March, April, 
July, and August, 1973. Rather than summarizing each of these reports 
separately, the following sections have reorganized the information for a more 
coherent presentation. 

II. ANALYSIS OF USER REQUIREMENTS 
A. Description of Target Audience 
The users of the proposed information system had been tentatively 
identified as a composite of six different groups of RDD&E personnel types; 
(1) sponsors, (2) performers, (3) trainers, (4) students/interns, (5) 
employers, and (6) practitioners. A brief description and an estimate of 
the size of each group, as well as Its subgroups, are presented below. 

ERIC 
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1. Sponsors 

Sponsors generally are personnel who make plans, policies, and decisions 
about dollar support for, or who directly influence the overall direction of, 
educational RDD&E personnel and training activities. Identified by 
institutional/agency affiliation, sponsors of educational RDD&E include 
persons in federal government, state and local education agencies, private 
foundations, industry and business, colleges and universities, and professional 
and academic associations. Figures developed by Gideonse (1969) on Fiscal 
Year 1968 expenditures for educational RDD&E indicate that, of the total 
documented minimum base of $192.3 million, 53% was provided through the U. S, 
Office of Education (USOE); 12% through the National Science Foundation (NSF); 
7^, through the Office of Economic Opportunity (OEO); 6% through the National 
Institute of Mental Health (NIMH); 4% through the National Institute of Child 
Health Developm^int (NICHD); 6% through other federal agencies; 4% through 
private foundations; and 8% through all other sponsors (state agencies, higher 
education institutions, and professional and academic associations). This data 
suggests that the largest group of user-sponsors are at the federal level. 
The transfer of some USOE responsibilities to the recently established National 
Institute of Education (NIE) will add the staff of NIE's Research Training Task 
Force to this group of federal RDD&E sponsors, and peripherally, the staff of 
other NIE task forces. 

Sponsors can also be defined by function and subdivided into groups 
concerned with (a) planning, (b) management, or (c) evaluation of RDD&E. The 
number of personnel directly involved in each of these subgroups is, at present, 
unknown. Hopkins (1971, p. 20) estimated a total USOE figure of 156 in 1966. 
Making allowances for the date of that estimate and extending it to include the 
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three functional groupings of professional personnel in other federal agencies, 
as well as paraprofessionals, Hood (Hood and Chorness, 1972, p. 4.9) suggested 
a total of 400 positions in the federal sponsor category. Adding program 
officers of private foundations, as well as sponsors in state and local 
educational agencies 3 the total number of sponsors probably does not exceed 
1 ,000 persons. 
2. Performers 

These are the individuals who implement the plans and policies conceived, 
approved, and/or supported by the sponsors. Personnel in this group can be 
categorized as either applied researchers or developers. Applied researchers 
are defined as personnel engaged in a "mission-oriented" production of know- 
ledge relevant to solution of a general problem. Developers, on the other 
hand, are identified as personnel engaged in design and production of content 
or training materials, or new training program or product models relevant to a 
speci f i c RDD&E personnel or training problem. Applied researchers, for 
example, might include senior authors of the American Educational Research 
Association (AERA) Task Force reports or the Oregon Studies in Educational 
Research, Development, Diffusion, and Evaluation (1972), as well as personnel 
affiliated with R&D centers. Developers include principal investigators, 
professionals, and paraprofessionals from the approximately 30 projects 
sponsored by the Research Training Task Force of NIE or selected personnel 
in regional laboratories, as well as personnel involved in a small number of 
development projects funded by other sources. The principal investigators 
probably number no more than 100 and their supporting professional and para- 
professional staff, a few hundred. 

Gene V. Glass and Blaine R. Worthen, Interrelationships Among Research and 
and Research-Related Roles in Education: A Conceptual Framework . Technical 
,9^- Paper No. 4. (Boulder, Colorado: University of Colorado, June, 1970) p. 9. 



3. Trainers 

This target population consists primarily of personnel who direct 
and instruct RDD&E training programs and also, by loose definition, 
personnel engaged either formally or informally in diffusion or linkage 
of the R&D personnel and training outputs. Trainers can be divided into 
two subgroups according to affiliation with degree or nondegree granting 
institutions and organizations. Byers (1971) surveyed degree-granting 
institutions throughout the United States. Out of 405 institutions 
responding^ she found 83 had RDD&E programs meeting the AERA definition, 
Byers also located and identified 55 nondegree granting agencies and or- 
ganizations (including 19 state education agencies, 9 R&D centers, 19 
laboratories, 7 private R&D organizations, and 1 USOE bureau) which 
regularly offer educational research-related training. Of 259 areas 
of emphasis in the 83 degree granting programs, only 10 were in "educational 
product or program development" and 4, in "dissemination and diffusion." 
The number of training directors and instructors involved in these programs 
might be estimated at several hundred regulars with major time commitments 
and several hundred more who occasionally teach a course or short-term 
institute. 

The diffusion-linking group is of potential importance, not so much 
because of its size, but because of its promise for stimulating interest 
in, and providing a channel for, delivery of decision-oriented, disciplined- 
inquiry training resources to educational practitioners. The projected 
proportional growth in diffusion positions anticipated by Clark and Hopkins 
(1959) has not been realized; we base this conclusion on available overall 
RDD&E position estimates (Hopkins, 1971), on evidence regarding training 
capacity (Fleury, 1968; Byers, 1971), and on demand and supply 
ERIC win and Worthen, 1970). 
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4. Students and Interns 

This group consists of students and interns who undertake R&D training 
programs on either a perservice or an inservice basis. The relatively 
slow growth of preservice training programs for paraprofessionals or 
technicians and undergraduates suggests that the number of users at this 
level of study may be small. 

At graduate levels, the preservice consumer audience is larger. 
Byers (1971) showed that, at the end of the 1970-71 academic year, 
there were 707 subdoctoral and 1,053 doctoral students enrolled as degree 
majors in educational RDD&E programs. Brezlnski and Smith, in their 
synthesis of ROD&E manpower supply and demand (1971), suggest that, in 
spite of the phase-out of the Title IV Graduate Research Training Program, 
a large number of persons are and will continue to be trained in RDD&E on 
a formal basis. Enrollment at 53 institutions responding to a questionnaire, 
they reported, included 443 persons at the subdoctoral level and 495 at 
the doctoral level. In addition, three recently funded RDD&E training 
consortia anticipate training approximately 225 persons at the doctoral 
and master's levels over the next three years. In sum, it appears that 
the preservice consumer audience in undergraduate and graduate degree 
programs will number at least 1,000 (per year) and possibly as 
many as 2,000-3,000 if "minors/' as well as "majors," are included. 

The number of inservice consumers- -RDD&E employees involved in 
continuing education or short-term training--is unknown. When the personnel 
training needs of a Dumber of existing R&D-related organizations and agencies 
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(e.g., regional laboratories, R&D centers, private, nonprofit R&D agencies, 
state education agencies, colleges and universities) are taken into account, 
however, the prediction made in the proposal for this project becomes quite - 
real. The proposal stated there will be "...a marked increase in the number, 
diversity, quality and duration of training [and, hence, in the number of 
training consumers] offered by nondegree granting organizations." A consumer 
audience appears to exist among presently employed or newly recruited RDD&E 
personnel who need to maintain awareness of current developments or to gain 
specialized knowledge and skill training pertinent to their work. This group 
appears to be potentially as large as, if not larger than, the preservice 
consumer audience. 
5. Employers 

This group is represented by the organizations. Institutions, agencies, 
and limited- term projects who hire personnel to perform RDD&E and related 
functions. Hood (Hood and Chorness, 1972, p. 4.8) estimated there might 
be 1,500 such active educational RDD&E projects receiving federal funding 
through USOE Cooperative Research or educational R&D sources in NSF, OEO, 
NIMH, and OCD. Because a single institution can receive support for several 
projects, however, the actual number of federally funded organizations should 
be estimated at only 1,000 to 1,300. 

Although this "core" of federally funded educational RDD&E projects 
certainly is the primary employer target, there are other employers of 
major interest, including: state educational agencies, local educational 
agencies, and nonprofit and profit-making R&D agencies operating with 
federal or other funding. In addition, there are substantial applied research, 
development, dissemination, and especially, evaluation project activities 
supported under other USOE funded programs (e.g.^ evaluation of ESEA Titles I 
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and III projects). Data supplied by Gideonse (1969), Clark and Hopkins (1969), 
and Hood (Hood and Chorness, 1972) suggest that nearly half of the total 
educational RDD&E funding may come from these other-than-"core" federal 
sources. 

State, intermediate, and local educational agencies who are not 
recipients of federal educational R&D funding constitute another major 
category of employers. It is difficult, however, to estimate the 
number of these who have an active or potential interest in educational 
RDD&E personnel and training information. Although there are approx- 
imately 19,000 elementary and secondary education systems and 2,600 
colleges and universities, only a small fraction of them employ even a 
few full-time educational RDD&E professionals.* This contention is 
supported by Bargar and Hagan's finding (1970) from a survey 
of Ohio public school administrators. Although these administrators 
were receptive to employing trained RDD&E personnel, they indicated 
that openings for such personnel do not exist, apparently because of a 
lack of funds or demand. Their responses indicated that 56% of 

large districts, 100% of medium districts, and 83% of small districts 
did not anticipate any openings in the 1969-1970 period for R&O trained 
graduates. Brzezinski and Smith (1971) contend that availability of 

For instance. National Educational Association survey data (Hood and 
Chorness, 1972, p. 4.9) shows only 30% of major school systems (LEAs 
enrolling 12,000 or more students) reporting one or more research ad- 
ministrator positions. Hopkins (1971) estimated 270 positions. NEA 
(1971) documents 321 positions in 137 school systems. Making allow- 
ance for paraprofessionals, Hood (Hood and Chorness, 1972) estimated 
the LEA full-time equivalent RDD&E positions as less than 500. With 
respect to colleges and universities, Hopkins' (1971) projections for 
1974 education R&D positions suggest approxin^ately 3,000 positions in 
educational RDD&E at colleges and universities in other than federally 
supported subunits (such as R&D Centers, ERIC Clearinghouses, Handi- 
capped or Vocational Education R&D Centers). 
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funding will be the critical variable affecting the demand for RDD&E personnel. 
The potential employer-market at the local level for RDD&E personnel and 
training information would also probably be affected by this same variable. 

A survey of RDD&E employers in tho San Francisco Bay Area (Hood, 
Banathy, and Ward, 1970) documented that there is another substantial area of R&D 
activity in business, industrial and nonprofit R&D agencies which employ 
personnel with RDD&E competencies and which are not supported by "core" 
federal funding. This employer sector is a competitor for educational 
RDD&E talent and a potential "customer" for educational RDD&F personnel 
and training information products or services. 
6. Practitioners 

Other than the lay public, educational practitioners are the most 
distant, but also the largest, group of potential users of educational 
RDD&E personnel and training information. Most practitioners are (or will 
be) employed by one of the classes of actual or potential RDD&E employers 
discussed above. It is conceivable to us that this group incTudeb all 
elementary, secondary, technical, and higher educaton teachors, administra- 
tors, and other staff, as well as educators v/orking in educational pro- 
fessional associations, foundations, etc. 

This population potentially will total several million people. As 
suggested in the project proposal, this entire population as a target 
represents a diffuse, hard-to-reach audience which may be largely beyond 
the scope and purpose of this information system's products and services. 
Isolating the smaller number of practitioners in direct RDD&E functions or 
positions is difficult, since according to Brzezinski and Smith (1971), 
reliable figures on the number of personnel currently employed in RDD&E 
do not exist. Much of this important subpopulation may, however, be reached 
Jiecause of overlapping memberships in one or more target groups mentioned. 
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B. Data on Information Needs of User Groups 
The description in section A of the information needs of the six potential 
u'^er groups for our system is, at best, a difficult task. Although substantial 
research on user information needs has been done in science, engineering, and 
psychology, among other disciplines, there is a general paucity of research on user 
information needs in education and specifically in educational RDD&E. Dershimer 
(1970, p. 1) reported that with the exception of "...two analyses of communica- 
tion networks made by Garvey and Lingwood, little precise information exis .s 
about the [communications] behavior of persons in educational research." Time 
and financial constraints on this project preclude the possibility of conducting 
an extensive survey of R&D information users and their needs. However, in- 
formation on user needs was collected from small but representative samples 
of high priority target groups as part of the pilot test of dissemination 
alternatives. 

1 . Purpose of the User Study 

To delimit design parameters for an RDD&E personnel and training informa- 
tion system, the project staff conducted a pi lot test of a prototype system. 
Feedback was sought from members of potential populations regarding their 
priorities for target user groups, information content areas, information- 
seeking purposes, and products and services of the system. 

2. Description of Sample 

Based on NIE priorities presented in the project's Interim Report of the 
Preliminary Design Phase, five types of target user populations were selected 
for involvement in the pilot test: sponsors, researchers, model/materials 
developers, degree trainers, and nondegree trainers. The samples from each 
population are described below: 

O 
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Sponsors . Although not representative of the full range of sponsors, 
five NIE staff active in researcher training and related projects were chosen 
purposively by the project staff as a central and significant sample. The 
group included staff from the Task Forces on Researcher Training and Dissemi- 
nation. 

Researchers , The lack of a substantial base of persons actually doing, 
or highly cognizant of, applied research on RDD&E training precluded the use 
of random sampling procedures to select this group. A purposive sample of 
five persons identified by the project staff as recognized experts in the 
field was therefore used. 

Model /Materials Developers . A simple random sample of nine was drawn 
from a list of three model consortia and 30 models and materials development 
projects funded by the NIE Task Force on Researcher Training, with the project 
director as the contact. Although this sample does not include other types 
of development projects, such as R&D programs of laboratories and centers, it 
does draw from the core of personnel actually producing RDD&E training. 

Degree Trainers . A 6% simple random sample (N=9) was drawn from the 
set of degree-granting programs reported in the 1971 AERA Directory of 
Training Opportunities in Educational Research and Related Areas . The contact 
person listed in the Directory was the proposed subject. 

Nondegree Trainers . A }S% simple random sample (N=9) was drawn from 
the set of nondegree granting programs again reported in the 1971 AERA 
Di rectory . The contact person listed in the Directory was the proposed subject. 

The samples for sponsors and researchers may inadequately represent the 
full range of interests of these two groups; however, the other three samples, 
although small, are true random samples of significant, enumerated populations. 
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3» Procedures 

The user study was conducted as part of the pilot test of dissemination 
alternatives (to be discussed in a later section of this report). 

Initial arrangements for participation in the product/service review 
were made through a telephone call to each contact person in the developer, 
degree trainer, nondegree trainer, and researcher samples. During that 
call, the purpose and procedures of the review were explained to the subject 
and an appointment for a follow-up interview within two weeks was scheduled. 
A package of materials was then mailed to the subject, along with a letter 
restating briefly the review procedures and confirming the date of the 
follow-up interview. Subjects were also asked to return by mall four response 
forms (similar to those previously completed by the sponsor sample) on which 
they indicated priorities for types of users, purposes, content, and outputs 
of the proposed information system. The subsequent telephone interview, 
requiring about 30 minutes to complete, asked subjects to indicate the 
utility and importance of each product and service. They were asked to ex- 
plain whether or not it would be useful and to supply other relevant informa- 
tion. 

4. Sunmary of Information Needed 

Information needs vary with user, time, purpose, location of information, 
and sources, among other factors. The following specification of types of 
information needed by the identified user groups is derived from analysis 
of the utility and/or desirability of information input to the specific user. 

Sponsors . The information needs of this target group are quite diverse. 
For the small number specifically concerned with personnel and training, 
nearly all types of information are needed: published and unpublished papers 
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and reports on RDD&E-rel ated policy, theoretical and practical issues; 
personnel supply and demand information, particularly personnel requirement 
trends and critical skills shortage areas; detailed information on the 
content, progress, and evaluation of sponsored training programs and pro- 
jects. For the larger number concerned with planning, management, or 
evaluation of other educational RDD&E, personnel and training information 
needs appear to be detailed only in aspects under the specific sponsor's 
purview. 

Interviews revealed especially the need for assistance in locating 
specific training practices, materials, or persons competent to conduct 
training in order to provide assistance to other agencies, such as, state 
departments of education and local school systems, and for similar assistance 
in conducting inservice training for USOE or NIE staff. Two individuals 
emphasized needs for personnel supply and demand information, with emphasis 
on details about available or potential supply and ability to perform. Two 
cited the need for information on validated RDD&E methodology or practice, 
especially how to diffuse these practices and provide for their utilization. 
Other needs mentioned included information on training facilities and 
capabilities and on the state of knowledge regarding personnel and training 
issues. 

Performers . Members of this group have a- wide range of information 
needs. There is general interest on the part of both applied rese^archers 
and developers in information on policy, trends, theoretical issues,* and 
research on training. Applied researchers have specific interests in 
RDD&E methodology and in descriptions of research and development on RDD&E 
training, as well as in analyses of RDD&E competency clusters. Developers, 
on^the other hand, have special needs for job and task analysis informa- 
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tion; RDD&E case studies; employment trend information, with details on 
skill, knowledge, and content shortage areas; directories of evaluated 
RDD&E instructi6nal materials and resources; and details on the content 
and nature of RDD&E training developments in progress. 

The heavy emphasis on the need for knowledge of all aspects of RDD&E 
instructional materials and program development was shared by all inter- 
viewees who were themselves currently engaged in R&D training materials 
development. All indicated a need for detailed descriptions of research and 
development on RDD&E training. Several cited a need not only for directories 
of instructional materials, but also for access to the materials. Need for 
evaluations of such materials was mentioned by two. Three other developers 
indicated a need to know what other projects were doing in development-- 
particularly in terms of guidelinesr objectives, and procedures for 
evaluation of products. One cited a need for information on funding trends 
and policy issues. General interests in various areas such as supply and 
demand trends, marketing, and job/task/competency analysis were also expressed. 

Trainers . The degree training director and nondegree trainers share 
general interests with developers and, to some extent, sponsors in policy, 
issues, funding and supply/demand trends, and developments affecting RDD&E. 
They appear, however, to be particularly in need of information on theoretical 
issues, methodology, and research on R&D. Like developers, they could also use 
directories of validated instructional materials and curricula and specific 
training aids, such as, outlines, case studies, compendia of readings, exercises, 
and competency tests. Personnel directories and precise information on employ- 
ment and internship opportunities for students are also special needs of this 
group. 
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5. Infonmation-Seeking Purposes 

Each informant was asked to allocate 100 points OTong several alter- 
native purposes for seeking RDD&E personnel and training information 
according to the priority the informant attached to that purpose. Table 
1 presents the averages for each group as well as a weighted average • 
(The basis for the weightings will be described in Chapter III), 

The weighted averages indicate that our 32 informants* allocated 
highest priority to seeking information regarding "results of related work 
performed by others." This is followed closely by "to keep current on 
general development' " Three purposes assigned almost equal priorities 
are "seeking information to develop alternative approaches to problems," 
"finding answers to specific questions," and "identifying relevant pro- 
cedures, methodology and materials." Next most important is "gaining 
theoretical information." Of lesser importance is "to keep current on 
workers in specific problem areas," "acquiring ideas for new work," and 
"developing comptency in an area by reviewing the state-of-the-art." 
Least important is the need for obtaining information to "develop educational 
materials." 

The sponsor group was unusual in specifying priority needs for speci- 
fic "other" information (e.g., prepare public information, evaluation 
reviews to judge quality of programs, provide basis for policy formulation). 
The sponsor group is also distinguished from the other groups by the low 
priority it attaches to gaining theoretical information. 

★ 

Everyone approached cooperated and only one person failed to accomplish the 
task. The 87% usable response rate is due mainly to persons who could not 
be contacted in time. 
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The researchers exhibit a pattern similar to the weighted average, 
placing greatest emphasis on "determining results of related work performed 
by others" and on "keeping current on general developments" and least 
emphasis on "acquiring ideas for new work," "identifying relevant proce- 
dures, inethodology, materials," and on "preparing educational materials." 
Researchers are most unlike developers and trainers in their higher priority 
for "determining results of related work," and their lower priorities for 
"finding answers to specific questions" and for "identifying relevant 
procedures, methodology materials." 

The priorities indicated by the two trainer groups are quite similar. 
The rank order correlation is .81. The only items differing by more than 
two points are that nondegree trainers indicate higher priorities for 
"developing alternative approaches to problems" and for "developing their 
own competency by reviewing the state-of-the-art." Degree trainers , however, 
place greater emphasis on "need for information in preparing educational 
materials." Overall, these are the major purpose priorities for trainers, 
in descending order: (1) "keeping current on general developments," 
(2) "determining results of related work performed by others," (3) "acquiring 
ideas for new work," (4) "finding answers to specific questions," and (5) 
"identifying relevant procedures, methodology, materials. 

The developer group exhibits a markedly different pattern of priorities 
on purposes for making Information requests. The rank order correlations 
are only .35 with degree trainers and .28 with nondegree trainers. There are 
two purposes for which priority differences are most apparent: developers 
place lower priority on "acquiring ideas for new work" and higher priority 
on "gaining theoretical information." These are the major purposes for 
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developers: (1) "identifying relevant procedures, methodology, materials," 
(2) "gaining theoretical information," (3) "determining results of related 
work," (4) "finding answers to specific questions," and (5) "keeping current 
on general develop;nents." 
6. Content of Information Sought 

Each informant was also requested to allocate 100 points among preferred 
categories of information content to show how important a particular category 
is. 

The average priorities for each target group and the weighted averages 
are presented in Table 2. We had anticipated that target groups would 
differ. Sponsors and degree trainers are least alike (Rho=.25) and developers 
and nondegree trainers are most alike (Rho=.87). 

Overall, the weighted priorities indicate that these are the five most 
Important content areas (in descending rank order); (1) "theory and concepts"; 
(2)"research on educational R&D"; (3)"policy, roles, and strategy"; (4) 
"methodology" and, (5) "training model descriptions." These five areas received 
over half of the weighted priority points. For sponsors , these same five 
content categories, in different rank order, are top choices, although ^'supply 
and demand" ties with "theory and concept" for fifth place. Developers also 
assign top priority almost to the same five categories, but place "training 
resource descriptions" fifth and "training model descriptions" sixth. 
Researchers are also consistent on the first four, but markedly downgrade the 
importance of "training model descriptions." Perhaps remarkable is that 
more than half of all the researchers/ content priorities are for just three 
of the fourteen content categories, namely: "theory and concept," "policy, 
strategy and roles," and "research on educational R&D." 
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Nondegree trainers allocate over half of their priorities to the five 
categories: "methodology," "theory and concepts," "program/project description, 
"research on R&D," and "policy, roles, and strategy." Degree trainers allocate 
their top points to "training model descriptions," "program/project descriptions 
"theory and concepts," "training resources" (the actual materials) and "assess- 
ment devices." 

Sponsors differ most from all other groups in that they alone attach 
importance to "supply and demand information" (it's the least important 
specified category for all other groups) and, conversely, the sponsors attach 
markedly less importance to "program/project descriptions." As noted 
previously, researchers are distinguished by the very high priority they attach 
to just three content areas: "theory and concepts," "policy, strategy and 
roles," and "research on educational R&D." Researchers place far less emphasis 
on "training model descriptions" and "assessment devices." 

Dfive lepers and degree trainers exhibit several moderate differences, e.g., 
developers set higher priorities on "methodology," "policy, roles and strategy," 
"research on R&D" and "training resource descriptions"; while degree trainers 
set higher priorities for "training model descriptions," "program project 
descriptions," and "other" content. As noted, developers and nondegree trainers 
were very much in agreement on priorities. The largest differences are found 
in the developers' higher relative priorities for "program/project descriptions. 

Degree and nondegree trainers differ on relative priorities for several 
content areas, with degree trainers attaching more importance to "training 
model descriptions," "training resources," "job market and placement informa- 
tion," and "other" information; while nondegree trainers indicate markedly 
higher priorities for "methodology" and "information regarding consultants 
and experts." 
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These data clearly suggest that an RDD&E personnel and training informa- 
tion center must be concerned with access to a broad range of information if it 
is to satisfy its various clients. There is some focusing, five 
broad content areas ("methodology," "research on R&D," "policy, roles and 
strategy," "training model descriptions," and "theory and concepts") account 
for half of the weighted priority. Every category drew at least a modest 
priority, from at least one target group, with no category receiving 
appreciably less than half the priority it would have received if priorities 
were allocated equally over all 14 specified categories. The only two 
categories of information content receiving less than 5% of the priority 
points from all of the five groups are descriptions of programs producing 
talent" and "descriptions of training opportunies." Some group believes 
they need information about nearly every one of the information content areas. 
Conversely, no content category received a predominant priority. "Theory 
and concept," the top weighted category, received 11.75% of the weighted 
priorities; it is also the highest priority for any of the groups, with 
researchers assigning it only an average of 19.32 of their 100 points). 

Hence, there is no hope of specializing with great emphasis on only 
a few information content categories of unusually high priority. The 
information needs of the target audience are just too diverse, both with 
respect to needed content and purpose. 

Discussion of the preferences of these five target audiences for 
specific informajfe-ion products and services will be found in .Chapter III. 
Before turning to that subject, we shall present further data on target 
. group information needs by resorting to data obtained by the NIE Researcher 
Training Task Forces 's R&D Training Information Ne e ds Survey, which was 
completed by 27 respondents attending the 1972 AERA presession on research 
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training projects. Our interpretation of these responses leads to the following 
conclusions. 

Students and Interns . The information needs of both preservice and 
inservice students and interns parallel! closely those of trainers and developers 
in the areas of instructional materials and resources, case studies, theoretical 
and policy issues, job and task analyses, and RDD&E methodology. Their needs 
are probably mone intense, however, for actual instructional materials, for 
specific information regarding R&D employment/internship opportunities, and 
for general information regarding education and career planning. 

Although only two of the respondents to the R&D Needs Survey considered 
themselves as representative of the student/intern user group, the group's 
perception of student/intern information needs still has some relevance for 
this analysis. However, even though respondents felt qualified to specify 
the types of infonnation students needed, they tended to vary more than 
the other user groups in their degree of need (high degree vs. low degree). 
The largest percentage of agreement among respondents«-74%--indicated a high 
student/intern need for periodic listings of training opportunites for various 
RDD&E competencies. About 85% indicated there was a need for case studies 
of RDD&E and about 81% indicated a need for copies of instructional materials/ 
packages. But in the former case, only 56% felt the need was "high" and, in 
the latter, only 51%. Again, about half the respondents also perceived that 
students would have "high" needs for papers on theoretical issues and on 
research in educational R&D. Fewer respondents (about 50%) saw high needs 
for personnel supply/demand information, especially for identification of 
programs which will produce various competencies, for statistical reports, 
and for analysis of competency clusters needed in various roles. Student 
infonnation needs were apparently perceived as fewer and less intense than 
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either trainers or developers— which suggests that the possibility that involve- 
ment by a trainer/instructor may have had an effect on the respondents' judg- 
ments. 

Employers and Practitioners . The employer group undoubtedly needs 
personnel supply and demand information, particularly in specific terms of 
available applicants or of institutions which can supply talent. There 
also appears to be an even greater potential need for information on relevant 
continuing education programs or instructional resources which could be used 
to support an employer's continuing education program. 

By contrast, the information needs of the practitioner group, appear to 
be much less specific, more diffuse, and general. This group's awareness of, 
and interest in, educational R&P may be reactive. Their needs, therefore, 
may have to be serviced only through provision of digested information on 
research and methodology of R&D, as well as on R&D products/instructional 
.materials and on personnel demand trends. 

Data verifying the information needs of both employers and practitioners 
is limited. In the R&D Needs Survey , R&D managers/employers/practitioners 
were identified as a single potential user group. The diverse interests 
represented in this one category may account for the variability in the 
responses. The two areas of "high" need were "identification of programs 
producing various competencies" (70%) and "papers on R&D policy issues" {6 %). 
Slightly less than 60% of the respondents perceived that this group of users 
would have high need for increased access to periodic listings of training 
oppor'tunities and for instruments to assess RDD&E competencies. About half saw 
users needing analyses of competency clusters for various R&D roles and 
evaluations of published instructional materials. The greatest emphasis was 
thus placed on needs for personnel supply and demand information, with a lesser 
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emphasis on general information about R&D policy and issues and about RDD&E 
instructional materials. 

C. Target Audience Requirements 

Table 3 summarizes the project's initial estimate of the sizes and probable 
needs of each major audience, as well as an assessment of the benefits or 
lost opportunity costs likely to be experienced by them. 

Table 3 is more prescriptive than descriptive regarding users' needs 
in that it lists what users should need (e.g., supply and demand information, 
job market/placement information) in addition to the common high priority 
content areas listed in the previous section (e.g., research on R&D, 
methodology, theory and concepts, policy, strategy and roles) if each class 
of users is to perform its roles more effectively. The table also suggests 
the benefits or lost opportunity (cost regret) which the presence or absence 
of such inforaation may cause. 

Table 3 

TARGET AUDIENCES FOR EDUCATIONAL 
RDD&E TRAINING & PERSONNEL INFORMATION 



TARGET ESTIMATED 
AUDIENCE SIZE 



Sponsors 400-1000 persons in 
planning, management 
or evaluation. Ap- 
proximately 20 persons 
directly involved in 
planning management 
or evaluation of RDD&E 
training. 



PROBABLE NEEDS, BENEFITS,. 
AND COST REGRET 



Adequate, timely information re- 
garding numbers and types of trained 
RDD&E personnel, supply, demand, pro- 
duction rates, skills, shortage areas, 
and state-of-the-art for delivery of 
training in specified RDD&E skill 
areas and subject content areas is 
essential for realistic planning and 
management, especially for large-scale 
directed R&D or practice improvement 
programs . 

Lack of accurate information may 
result in misallocation of resources, 
choice of inefficient or ineffective 
strategies and approaches, program 
drift, delays or failures in achieve- 
ment of objectives. 



TABLE 3 (CONTINUED) 
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TARGET 
AUDIENCE 



RDD&E In« 

structional 

Materials/ 

Program 

Developers 



ESTIMATED 
SIZE 



50-100 principal 
investigators , 
several hundred 
performers. 



PROBABLE NEEDS, BENEFITS, 
AND COST REGRET 



Information regarding personnel 
trends, skills shortage areas and 
newly emerging needs can create aware- 
ness and capability to propose and 
develop needed training materials/ 
programs. Specific information re- 
garding required competencies, work 
contexts, levels of skills, etc. is 
needed for efficient design and de- 
velopment of curricula and instruc- 
tional materials. IPiFonnation 
regarding instructional materials and 
methods now available or in development 
elsewhere can increase effectiveness 
and reduce costs of developments. 

Lack of information may result in 
failure to respond to critical training 
needs, in duplication of effort, or in 
programs or materials inefficient. 

Inability to provide for dissemi- 
nation of Information regarding train- 
ing developments may result in under- 
utilization by intended targets. 



RDD&E Train- 80-120 degree program 
ing Program directors; 80-120 non- 
Directors/ degree program heads; 
Instructors several hu.idred 
instructors . 



Information regarding personnel 
trends, skills, shortages, emerging 
federal program support is needed to 
justify and establish new training 
programs or to modify existing ones. 
Instructional materials catalogues, 
directories of training consultants 
and curriculum development aids are 
needed to quickly and efficiently de- 
velop new programs or modify existing 
ones. Information regarding employ- 
ment and intern opportunities and 
trends is needed to counsel and place 
students and interns. Access to case 
studies, published and unpublished 
papers and reports, and knowledge of what 
other programs are doing can stimulate 
innovation and renewal of program con- 
tent and method. 
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TABLE 3 CCONTINUED) 



TARGET 
AUDIENCE 



ESTIMATED 
SIZE 



PROBABLE NEEDS, BENEFITS, 
& COST REGRET 



Lack of information can result in 
these consequences: serious misdirection 
of training resources resulting/ in over- 
supply or shortages of trained talent 
yiLl vil the. employment market; added 
retraining costs or inefficient use o. 
personnel by subsequent employers; 
failure to attract or hold highly 
qualified candidates; failure to place 
students in effective internships or in 
appropriate jobs and; inefficient use of 
training and educational resources. 



Students 1750-2000 persons in 
and degree programs, pos- 

Interns sibly 1000-2000 more 

in significant "minors 
several thousand more 
full -time employees 
in continuing educa- 
tion. 



Many needs of the student/intern group 
may be met indirectly through the training 
program/instructor group. The student grou 
will especially profit from the availabilit 
of information on programs and materials an 
of intern opportunities that will prepare 
them both for inmediate and long-term em- 
ployment in RDD&E positions- Information 
concerning job and training opportunities i 
of particular interest to the person consid 
ering enrolling in an RDD&E program. Job 
opportunities are of soecial interest for 
those seeking full or part-time employment. 

The full -time employee in a con- 
tinuing education program is specially 
interested in where and how he can up- 
grade his competencies. 

Benefits and cost regret for 
trainees are similar to that for 
programs. 
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TABLE 3 (CONTINUED) 



TARGET 
AUDIENCE 



ESTIMATED 
SIZE 



PROBABLE NEEDS, BENEFITS, 
& COST REGRET 



RDD&E 1000-1300 federally 

Employers funded projects, sev- 
eral hundred larger 
local education agen- 
cies, possibly a 
1000 colleges and 
universities, several 
hundred R&D agencies, 
and organizations. 



Employers need general information 
regarding supply and demand in order to 
effectively plan and manage their R&D 
efforts. Benefits and cost regret are 
similar to federal sponsors though usu- 
ally on a much smaller scale. Knowledge 
of job applicants or programs producing 
applicants, especially in critical skills 
shortage areas can reduce recruitment and 
training costs. Knowledge of effective 
training programs or validated materials 
that can be made available to employees 
can reduce training costs and raise 
efficiency. 



Practi- May include several 

ti oners million teachers, 

administrators, and 
other professionals, 



General knowledge regarding educa- 
tional RDD&E may create awareness, in- 
terest and willingness to support 
Investment In and readiness to utilize re- 
sults of educational R&D, RDD&E students 
and performers may be recruited from this 
population. Many members of this target 
group are potential part-time users of 
decision-oriented disciplined inquiry 
or are decision-makers regarding utili- 
zation of R&D products. 

Benefits and cost regret of 
RDD&E training information peir se are 
hard to estimate > but may be relaii vely 
much less important than for other 
target groups. 
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III. PROMISING ALTERNATIVES FOR THE DISSEMINATION OF 
EDUCATIONAL RDO&E PERSONNEL AND TRAINING INFORMATION 

As part of a pilot test of alternatives for disseminating information about 
RDD&E personnel and training, samples from five potential target user groups were 
contacted to evaluate fourteen possible information products and services. These 
products and services, the pilot test procedures and sample selection, and the 
overall results of the user evaluation are briefly described in the first section 
of this report. A second section presents an analysis of the purpose, rationale, 
user evaluation, and cost estimates of the top rated dissemination alternatives. 
Conclusions and recommendations are contained in the surranary section. 

A, Pilot Test 

1. Selection of Products 

The project staff and consultants considered a large number of alternatives 

for dissemination. Given the general character of the emerging design for a 

proposed information system, they were nearly unanimous in nominating several 

products and services (i.e., announcement brochure, newsletter, inquiry services) 

as essential. A second set of products or services survived screening as possible 

alternatives. Fourteen dissemination alternatives were finally selected for 

1 

inclusion in the pilot test. These are grouped below by type: 
Announcement & Current Awareness 

Personnel and Training Newsletter 
Personnel and Training Abstracts 

Services 

Hotline Query Service 
Mail Query Service 
Walk-in Query Service 

T 

The selection of products is described more fully in the Interim Report of the 
"g^-sct's Preliminary Design Phase (Hood and McCutchan, December, 1972, Chapters 
ERiC"^ III, and Appendix D). 
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Infonnal Communication 

Conferences and Meetings 
Telecommunication 

Products 

Instructional Materials Catalog 

Grant Package 

Mini-Collection 

Case Study Source Book 

Literature Source Book 

Directory of Training Opportunities 

Dissemination and Utilization Service 

Technical advice on "marketing,*' copyrights, and publication 
of instructional materials. 

2, Selection of Samples 

Based on NIE target group priority data (Hood and McCutchan, December 1972, 
Chapters I, III, Appendix B), five target user populations were selected: 
(1) NIE/USOE Staff (sponsors); (2) NIE/Research Training-sponsored Project Developers 
(performers); (3) Degree Training Program Directors; (4) Non-Degree Training Program 
Directors; and (5) Researchers (on educational R&D personnel and training). 

Given the absence of appropriate sampling frames for the sponsor and researcher 
populations, five NIE/USOE staff and five researchers, each with several years of 
experience in the field, were selected. Eight supplied usable data. For the 
performer, degree program director, and non- degree program director populations, 
nine persons were selected randomly from each of three lists: (1) a list of recent 
and currently funded Research Training Project directors, (2) a list of directors 
of degree-granting educational R&D training programs compiled from the 1971 AERA 
Directory of Training Opportunities , and (3) a list of directors of non-degree 
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training programs also compiled from the 1971 Directory , Eight of the nine 
developers, seven of the nine degree trainers and eight of the nine nondegree 
training directors provided usable information. The total number of usable 
responses was 31 (out of 37 selected). 

3. Procedures 

Each person was contacted by telephone (or in a few cases, visited personally). 
That initial contact was used to explain the information design project and to 
request assistance in the pilot test. A package of materials containing descrip- 
tions of each of the 14 dissemination alternatives and a response form was then 
sent. Approximately two weeks later, each person ms contacted again and inter- 
viewed over the telephone. Each person then completed and returned the response 
form on which 100 points were allocated among the 14 alternatives in terms of 
priority preferences. 

4. Results 

Table 3 presents the summary point allocations for the seven highest priority 

alternatives. Each unweighted average represents the simple sum of points assigned 

by the 31 respondents to an alternative divided by 31. The weighted average was 

2 

obtained by weighting the five averages for the five target groups, summing these 

weighted averages, and then dividing by five. 

The weighted and unweighted averages are quite similar. In both cases, 

3 

the top seven (of 14) alternatives received 68% of the points. The two 
— 

The weights were obtained from information supplied by each respondent regarding 
his priority allocation for serving each of 16 target sub-groups. Points assigned 
by the five sub-groups to the same five sub-groups were computed, rescaled, and 
used as initial weights which were applied to obtain a new set of weighted averages. 
The weighted averages were in turn summed to compute new weights. After several 
iterations, the weights stabilized at: sponsors, .231; developers, .195; degree 
trainers, .214; non-degree trainers, .143; and researchers, .217* 

^An analysis of variance was performed using an unweighted means analysis for multi- 
factor (dissemination alternatives and target sub-groups) experiments having repeated 
measures. The results indicate that there is a significant alternatives-by- target- 
groups interaction (P<#05), indicating that groups do differ in their priorities; 
but the main effect for alternatives is highly significant (P<. 005) when tested 

i^y^inst this significant interaction term. 
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TABLE 4 

PRIORITY POINTS AWARDED TO THE SEVEN TOP DISSEMINATION ALTERNATIVES 







Weighted 
Averaae 


Unweighted 
Averaa@ 


1. 


Newsletter 


14.05 


15.52 


2. 


Annotated Listing of Current Literature (Abstracts) 


12.15 


. 11.35 


3. 


Literature Source Book 


10.41 


9.28 


4. 


Instructional Materials Catalog 


9.02 


8.99 


5. 


Hotline 


8,20 


8.02 


6. 


Answer Mail Inquiries 


7.39 


7.86 


7. 


Case Study Source Book 


6.99 


7.34 




Total (of 100 possible points) 


68.21 


68.36 



announcement-and-current-awareness items. Newsletter and Annotated Listing of 
Current Literature (or Abstracts), topped the list and accounted for over one 
fourth of the priority points. Two of the services, the Hotline and the Mail Query, 
ranked fifth and sixth respectively. The three top products were the Literature 
Source Book (third), the Instructional Materials Catalog (fourth) and the Case 
Study Source Book (seventh). 

B. Description of Top Dissemination Alternatives 

The description, rationale, user study documentation, and cost estimates for 
these seven dissemination alternatives are described below. 
1. Announcement and Current Awareness 

The following two items have the general purpose of "broadcasting" information 
to targeted audiences: 

Personnel and Training Newsletter . The major purpose of the Newsletter is 
O to provide key target audiences (especially NIE/USOE staff. Researcher Training 
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contractors/grantees, other RDD&E training "perfonners" engaged in program planning 
or evaluation, research, development, or diffusion, and training program directors) 
with concise, current news ajid information about personnel and training. 

The rationale for a Newsletter is based on the practical experience of mny 
information centers that no other single activity is as cost effective for 
creating and maintaining awareness of the existence and capabilities of an 
information center. If a Newsletter is well written and targeted to the interests 
and current information needs of its readers, it will catch the attention of 
individuals otherwise too busy to attend to specialized information. Since 
personnel and training will be a "part-time" concern of the great majority of 
the members of any of the target audiences, the Newsletter provides an opportunity 
to call their attention to information items which otherwise might go unannounced 
or unnoticed. 

The content of the Newsletter may include: announcements of RFP's, awards, 
reports, meetings and conferences; substantive descriptions of major policy 
papers, research reports, case studies, and materials developments; and solicited 
articles on R&D personnel and training needs, issues, methodology, and models. 
It also may regularly announce special services and products of the information 
center. 

The test format of the Newsletter consisted of three issues (November and 
December 1972 and April 1973) each between four to nine pages, standard 8 1/2" 
X 11" paper, typewritten, single-spaced copy, offset press. Copies of the two 
1972 issues were mailed to each of the 31 pilot test respondents; all three issues 
were dist^^ibuted to 150 other target users. 

Personnel and Training Abstracts . The purpose of this product is to call 
attention to publications available through ERIC, National Technical Information 
Service (NTIS), Psychological Abstracts , and other collections which might be of 
ir O t to the various target audiences. 
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The rationale for this product rests in documented findings (Fry, 1972; 
Wanger, 1972) that relatively few ERIC users routinely scan Research in Education 
or the Current Index To Journals in Education^ Moreover, we suspect that few 
members of the target audiences have access to or would bother to search the NTIS 
Weekly Government Abstracts, Psychological Abstracts , or similar sources. Some 
kind of product providing current awareness of new literature is desirable. A 
selective dissemination of information (SDI) format was considered, but rejected 
because: (1) the subject of educational RDD&E personnel and training is already 
a relatively narrow content area and (2) the cost of developing even a few group 
interest profiles and publishing separate SDI packages would be much too great. 

The Abstracts use a simple subject category arrangement to facilitate quick 
scanning of contents. The content of the Abstracts represents current selections 
on educational RDD&E personnel and training from RIE, CUE, NTIS, and other abstract 
or citation publications. The selections are organized by relevance to various 
educational RDD&E personnel and training topics. Each selection is listed by title, 
accession number, author, date of publication, length, and cost of microfiche 
and/or hard copy. In addition, an abstract or annotation is presented with each 
selection. 

Abstracts have been designed as a separate product which may be mailed with 
the Newsletter at no great additional cost. Its test format paralleled that of 
the Newsletter. One issue was mailed to a large user group with the first Newslette 
and a copy was also included in the package mailed to each of the 31 pilot test 
respondents. 

The user study results show the two current awareness items ranked first and 
second, jointly receiving over one fourth of all the points assigned. Although 
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the sponsor group assigned the Newsletter a remarkably low average of 2*75 points, ^ 
all four of the "field-based" groups--developers, degree- trainers , non-degree 
trainers, and researchers— gave the Newsletter top priority. They saw the 
Newsletter as a quick and easy means of keeping informed about general developments 
and current activities in the field of RDD&E personnel and training* Developers 
commented that it *'...is a convenient and efficient way to keep up to date" and 
"...gives an Idea of current status in the field-" Others saw it as "political" — 
it tells them "who's doing what" or gives them "leads on hot projects or for any 
project I might be working on." Degree trainers shared similar perceptions about 
the convenience and efficiency of the Newsletter and were able to relate it to 
their own planning needs--perceiving it as a means to get ideas for training 
or as a source of names and projects to contact. Nondegree trainers mostly saw 
the Newsletter as a useful awareness device to "...gain information about what's 
going on in R&D." One researcher noted that there simply was no good source of 
information about personnel and training. Another researcher gave it top priority 
because of its very high "utility" and "importance" in providing screened, current 
information in a manageable form. 

Regarding the annotated listing (Abstracts) of current literature, several 
persons commented that it overlapped in content with the monthly abstracts in 
Research in Education , but might be useful to people who do not review RIE 
frequently. Several developers and trainers found it a "succinct," "handy" 
way to keep informed, with several developers suggesting that it be combined 
with the Newsletter as a regular feature. Respondents who rated the Abstracts 
as "not useful" said they didn't need such a service because they review RIE 
and CUE regularly. 
4 

The sponsors may enjoy a surfeit of current information such as that contained 
in the Newsletter and thus may not see much value in the Newsletter serving 
O information needs. 
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Cost Estimates . The cost of publishing either the Newsletter or the Abstracts 
will depend greatly on the accessibility of information, since most of the cost 
is associated with securing, selecting, and preparing copy, rather than with 
reproduction and mailing. If the Newsletter is to be current and broadly based, 
its editor(s) will have to actively seek out information from a wide variety 
of sources. 

Experience in preparing three issues of the Newsletter suggests that approxi- 
mately five days of professional ''editor" time are required for correspondence, 
telephoning, writing, and editing an issue. Another two to three days of clerical 
time are required for typing, proofing, addressing, updating mailing lists, etc. 

Preparation of the Abstracts may require two to three days of professional 
time on searching and selecting items and on writing or rewriting item annotations. 
The user study information clearly indicates the need for "rewriting" abstracts to 
highlight personnel and training implications. 

The major (in roughly descending order of cost) variables are: (1) the number 
of issues per year, (2) the "quality" of infonnation (how recent, how useful, 
how well selected, how well edited), (3) the number of copies per issue, (4) 
the number of pages per issue. These are based on tne assumption that the 
Newsletter would have a format similar to the three pilot test issues, with a run 
of 250 to 1,000 copies. 

Applying Far West Laboratory salary, benefits, and G&A (overhead), we arrive 
at an estimate of approximately $500 to $ftOO per issue. This estimate includes 
all costs to the point of camera- ready copy for a six-to-eight page, "high 
quality information content" issue. Since reproduction and mailing costs may run 
between 20 and 30 cents per copy, a mailing of 400 copies per issue may add less 

5 

Fir<^t class ^nailing at 8t per item may be advisable. Third class mailing requires 
a third class permit at annual cost of $30.00, a one time $15.00 permit to impnnt. 
and a 4.8(t per piece rate if maximum weight per item is just under three ounces. 
Mailings would have to exceed 1500 items in the first year (e.g., six issues mailed 
FRir to a list of 250 addresses) to beat the first class rate- If multiple copies are 
q«nt to the same address, the cost may be closer to 15 to 20 cents per copy. 
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than $100 to the cost per issue. How many copies would be needed? Our rough 

guess is that 50 copies (bulk to NIE address) for NIE staff, 50 copies for sponsored 

performers, 200 copies for training program directors, and 100 copies for R&D 

employer organizations would constitute the initial mailing of 400. This mailing 

list will grow, but, if controlled. It might be kept under 1,000 "key" recipients. 

The cost for reproduction and mailing might then be estimated at approximately $250 
6 

per issue. If the Abstracts is included in the Newsletter (estimated at two to six 
more pages per quarterly or bimonthly issue), approximately $250 to $350 should 
be added to the cost per issue and three to five cents per copy, to the mailing 
cost, 

Siven the above inforrriation, we estimate that a quarterly Newsletter plus 
Abstracts publication (totalling an average of 10 pages per issue), mailed to 
no more than 1,000 non-paying subscribers, would cost approximately $4,000. A 
bimonthly publication should cost less than $6,000. 

It must be noted that over half the cost of publication is represented by 
professional news collection, writing, and editing. This portion could be cut 
substantially, but at the risk of losing overall quality of information content 
and potential utility to the reader. The choice of a well-informed, experienced, 
and dedicated editor is essential, "Journalism" skills are necessary, but not 
sufficient. The editor must know what information the various target audiences 
need and how to obtain and organize it for their current awareness. 
2. Products 

Of six products listed, two, possibly three, appear to be promising alterna- 
tives in terms of user evaluations. The top contender is a Source Book of Selected 
Literature, closely followed by an Instructional Materials Catalog. In third 
place, as a possible, is a Source Book of Case Studies. 

^A nominal charge of ono dollar per year, payable in advance, would possibly just 
Q '^r the cost of adding one addressee. Such a device may be needed to keep the 
:D!r ing list within the budget. 
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Literature Source Book , The purpose of this item is to provide trainers, 
training developers, students, and RDD&E practitioners with convenient references 
to literature which might be useful as basic or supplementary readings on the 
issues, concepts, research, methodology, and actual practice of educational 
RDD&E. 

The rationale for this product is found in the fact that both trainers and 
training developers report difficulty in locating instructional resources outside 
the more traditional "educational research" areas. A Literature Source Book could 
be used by trainers, students, and RDD&E practitioners to update their knowledge 
of what is actually happening in RDD&E. This is not such a problem for the 
researcher; however, the dearth of published literature with instructional value 
relevant to educational development, diffusion, and evaluation and allied, 
decision-oriented ^'disciplined inquiry" areas is a major problem. The state-of- 
the-art is neither well conceptualized nor well comnunicated in these areas. 

The content of this Source Book would include: (1) an introductory section, 
describing content, suggesting uses, and providing instructions on how to obtain 
the references; (2) a "deep" subject index, oriented especially to instructional 
or self-study requirements; and (3) a file of references, complete with citations, 
abstracts, and evaluative annotations. 

The test format was a semi-prototype. This Literature Source Book contained 
36 references and abstracts, a table of contents, and an index. 

In the results of the pilot test, the Literature Source Book received a 
weighted average of 10.41 points or third place (after the Newsletter and the 
Abstracts) . Target groups were markedly different in their evaluation of this, 
alternative, however. Sponsors averaged 14.43 points; researchers, 13.75; degree 
trainers, 9.3d; non-degree trainers, 8.75; and developers, 4.29. Neither the 

ERIC 
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developers nor the trainers were very specific about the uses to which they would 
put the Literature Source Book. They perceived it as a "nice," but not "essential" 
item to have. Developers commented that it could "help in a general way on a 
specific problem" and that it "deals with theoretical and conceptual areas of 
information and is generalizable." One degree trainer would use it to "find out 
about how people in DD&E are proceeding--developing a whole new jargon which 
[they] claim is different from research but may not be." Another would use it 
to "spur on the thinking of faculty." Nondegree trainers compared the Source 
Book to the Newsletter as a way to keep informed, but in a more confined content 
area. They considered it a "basic document," "more useful than the abstracts," 
and "an aid in doing [their] own research." The researchers indicated that they 
would use the Source Book as a starting point in defining the problem, in looking 
for clues to find authors and sources, and as a general source for skimming to 
keep current on who is writing what. 

Cost estimates for the Source Book reflect t!ie fact that the project has 
produced over 80 literature references and abstracts, principally based on the 
contents of the Oregon "Compendium." Hence, there is already a useful base for 
this product. Developing these items to publishable quality, including expert 
review and revision, and preparing camera-ready copy might cost $1,600. Our 
estimate is that each additional new entry would cost $75 to $100, including 
costs to search, acquire, screen, abstract, index, and format the draft item, 
submit it to at least two expert reviewers, and revise as needed. 

A 200-item Source Book, complete with index, might cost approximately $11,000 
to prepare for camera-ready copy. Depending on the type of reproduction and 
binding and the number of copies, production costs might run from $1.92 to $2-50 
per copy. A thousand copies could be produced by GPO for less than $2,000. 

ERLC 
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Note that one pilot test respondent suggested that a loose leaf format for the 
Source Book would permit updatings, reorganization of contents, copying, etc. 
Such a format could increase the bulk of the Source Book, but might be worth the 
inconvenience to gain the advantage of flexibility. Printing costs for a loose 
leaf, three-hole punch format are estimated at $2.60 to $3.50 per copy. A 
thousand copies could be produced by GPO for less than $2,700. 

Instructional Materials Catalog . The purpose of this product would be to 
list, describe, and evaluate selected instructional resources and materials which 
might be of value as resources for training programs and training development 
efforts. 

The rationale for this product, like that of the Literature Source Book, 
is found in the fact that both trainers and training developers report difficulty 
in locating instructional materials outside the more traditional "educational 
research" areas (e.g., statistics, experimental methods, tests and measurements). 
The problem is especially serious for "thin market'* items (especially those 
involving multiple media) which have failed to obtain a publisher, or, if 
published, have not received effective promotion. 

The test format was a "prototype" Catalog containing 36 entries (43 entries 
in a later version) and an index. 

Results of the pilot test show that the Instructional Materials Catalog 
was the fourth highest priority choice, receiving an overall weighted average of 
9.02 points. Sponsors gave it an average of 11.70 points; developers, 11.43; 
nondegree trainers, 9.38; degree trainers, 6.50; and researchers, 6.25, Sponsors 
and developers rate the Catalog high as a dissemination vehicle for creating 
awareness of available training products. The seven developers would use it as 
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a resource if they were "developing training" or "looking for information" about 

existing training materials. Trainers, on the other hand, would use the Catalog 

to "keep up-to-date," to J^elp put together "inservice workshops for teachers 

and supervisors in schools," or to "identify" curricular materials. One developer 

and two trainers also indicated that the Catalog would be more useful to them if 

descriptions of the materials were "more evaluative" or "critical." 

Estimating the cost for this product is quite difficult, simply because 

so much experience with the development of the ALERT system has led us to realize 

how expensive it can be to secure adequate, user-oriented information on any kind 

of educational development product. Two developers, both highly experienced with 

the ALERT system, independently arrived at similar cost estimates--approximately 

$500 per development product —to obtain information of similar quality to the 

ALERT Sourcebook of Elementary Curricula Programs and Projects (GPO, 1972). 

This may set a high q'»ality, upper limit. Assuming that any kind of independent 

analysis and e ;al«,ation is involved, $100 per item aiay be a credible lower limit. 

How can this very wide $100 to $500 range be explained? Much depends on the 

following factors: (1) how selective the collection will be, (2) how much 

searching will be conducted and in what areas, (3) how much analysis and evaluation 

will be undertaken, (4) how large the collection will be (economy of scale). If 

* Books in Print , the NICEM Indexes , the Westinghouse Catalogue, and similar sources 

are used, it is not hard to find published materials. Locating unpublished 

materials is more difficult and finding materials not in traditional research areas 
8 

is most difficult. 
7 

A development product may consist of several related items. Much of the cost 
is associated with locating developments, corresponding with developers to 
obtain information, etc., which often involve similar effort whether one or 
more development items are involved. The per item cost should thus be somewhat 
lower. 

8 

Guba and Gephart (1970) provide a most pertinent account of their efforts to 
' V> '-e fugitive training mate':"ials. Through presentations at meetings, involving 
g[^Qrsons, visits to 47 institutions, and over 800 mail contacts, they identified 

(cont.) 
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Exactly what the cost might be depends heavily on what kind of an Instructional 
Materials Catalog is desired. A "mix'* might be desirable— with approximately 200 
entries represented by a carefully evaluated selection of published texts, films, 
and other instructional resources and with 100 items represented by ALERT-like 
quality developments. Assume that all these items have been evaluated by* at 
least two independent reviewers or qualified users. Assume, moreover, that no 
more than 150 of the items would be pemiitted in the traditional research areas. 
Assume, finally, that at least five experts have received the draft copy before 
final revision. Delivery of camera-ready copy for such a Catalog would probably 
cost between $75,000 to $100,000. 

On the other hand, a simple listing of 300 easily located RDD&E instructional 
resources, with perhaps a one-paragraph description per item and a simple index, 
might be produced at a cost of less than $10,000. We doubt, however, that this 
alternative is a bargain. Most of our target group users know how to search 
for published materials and many are inundated by publisher brochures. These 
users want evaluation and critical review. They want help in identifying the 
"hard-to-find." They want adequate indexing, description, and analysis so 
they can match what's available with their needs. And they want adequate 
references to what is actually available. This kind of information is hard 
to retrieve. It costs more. 

Case Study Source Book . The purpose and rationale for the Case Study Source 
Book are quite similar to those for the Literature Source Book, namely, to provide 
annotated references to selected instructional resources. Case studies represent 

8 (Cont.) 

327 different items. Over three-fourths were textual materials or examinations. 
Less than 7% of the items (or 25 to be exact) dealt directly with development, 
diffusion, or evaluation processes. The proportion of DD&E materials should have 
increased somewhat in the intervening six years (Spring, 1967 for the first mail 
survey), but our own search for DD&E, as opposed to "research" (including 
statistics, test and measures, experimental design, sampling, etc.) materials, 
suggests that searching for the relatively rare and highly fugitive item can be 
costly. 
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an important resource for understanding the context and substance of RDD&E, which 

are often lost in the literature published in journal articles, monographs, and 

f*;rTnal reports. Decision-oriented "disciplined inquiry" rests heavily on experience 

9 

and judgment. Where this is true, "case methods" of instruction are indicated. 

The content of this Source Book would include: (1) an introductory section, 
describing content, suggesting uses, providing examples of such uses, and giving 
instructions on how to obtain the actual cases referenced, (2) a "deep" subject 
index, oriented especially to instructional or self-study requirements, and (3) 
a file of references complete with citation, abstract, and evaluative annotations. 
The test format was a semi -prototype, containing 41 references and abstracts- 

In the results of the pilbt test, the Case Study Source Book placed third on 
the list of six products and seventh on the list of 14 products and services. 
Receiving an overall weighted average of 6.99 points, it is most valued, as one 
would suspect, by the trainers and least valued by the researchers and developers- 
The group averages were: non-degree trainers, 9.38; degree trainers, 8.13; sponsors, 
7.90; developers, 5.14; and researchers, 5.00. 

The developers' uses for the Case Study Source Book differed from those of 
the trainers. The fomier thought the Source Book would have some utility when 
they were "looking for specific methods or materials" on RDD&E training development. 
Some would prefer to see case studies on training, as opposed to curriculum, 
development projects so they could acquire more detaileJ information about process 
and methodology. Both degree and non-degree trainers, on the other hand, perceived 
the Source Book as mostly useful for stimulating thinking about faculty research 
and for teaching courses in evaluation and educational research. Neither developers 



9 

Professional training in law, business administration, and systems analysis are 

-'^'"ples where study, analysis, and discussion of case materials are used to 

ER^O*^ professionals for decision-oriented inquiry. 
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nor trainers thought they would U5e the Source Book regularly and frequently, but 
both thought they would have "some uses'* for it, as indicated above. The 
researchers had various reactions. One considered the Source Book examples as 
too cryptic, only a teaser. Two researchers noted that the case studies typically 
were too shallow or superficial and that more effort would be needed in abstracting 
the substance. The Source Book should make a. greater effort to spell out what the 
reader would or could find in the case study. Generally, the researchers, although 
recognizing the value for training, did not see the case studies as being 
especially important to them— except when examining a specific domain of activity. 
Then, case studies, if well done and specific, could b6 especially useful in 
showing the real world or in gaining insight into the context and process of 
RDD&E. 

The Case Study Source Book may be a special situation as far as cost is concerned. 

There are 41 case studies abstracted in the prototype. Our guess is that there may ! 

be no more than another 10 or 20 case studies of comparable quality in existence 
10 

at this time. Moreover, there does not appear to be any likelihood of an 
appreciable number appearing in the near future. If this guess is correct, the 
existing prototype Source Book represents a substantial start toward a publishable 
product. The abstracts do need reworking, to indicate better what in the cases 
might be of use to different target audiences. Further indexing is also needed. 
Probably most important for actual utilization is the need to add a section 
describing and illustrating how the case materials might be used. A quick "clean-up" 
of the prototype Source Book might be accomplished for a few thousand dollars, 
depending on how much is attempted. 



On the other hand, there are literally thousands of R&D project reports. These 
reports might be "mined" for possible use in developing case materials. Alternative 
one might attempt to stimulate the production of case studies, possibly by providing 
incentives to selected projects to document their history in a case study or by 
Q commissioning contractors to produce more studies such as those produced by the 
ERXC O^^Son studies or AIR. 



52 



A more extensive effort might deserve consideration. The scope of work could 
include: (1) further searching to locate at least 50 case studies, (2) development 
of a thesaurus of case study subject terms, (3) analysis and in-depth indexing 
of each of the 50 selected case studies, and (4) preparation of at least two 
introductory chapters, one describing the case studies and their uses in policy 
study, program planning, research on personnel and training, training and personnel 
development, etc., and the other illustrating various applications of case materials. 
The body of such a source book would contain approximately 100 pages of substantive 
abstracts of the 50 case studies. Finally, there would be an in-depth index having 
some of the character of a concordance, in that it would attempt to identify all 
selected topics and the specific location in each case study of relevant material. 
This kind of a product might require five to seven professional personnel months 
and cost between $12,000 and $15,000. 
3. Query Services 

Three inquiry services were separately identified: (1) Hotl fne /Telephone 
Query, (2) Mail Query, and (3) Walk-in Query. Although the three services have 
basically the same purpose, which is to provide immediate, individualized response 
to specific user queries, they differ in the mode of incoming contact with the 
user. 

The rationale for providing thene services is simply that direct response to 
individual users is essential if effective, cost-beneficial service is to be 
provided at all. The analysis of user requirements (report of the Planning Phase, 
Chapter III) indicates that most target groups are relatively small and that 
they have diverse needs for information or information referrals. For some groups 
(especially sponsors and performers), if the information is to have value, it must 
be timely and relevant^ Moreover, direct contact with individual users is essential 
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to the concept of a user-oriented information center, since direct contact will 
enable immediate and continuing feedback concerning the adequacy of its collection, 
trends in information needs and interests, and user's evaluation of services and 
products. It may be expected that nearly all of the query load will come over 
the phone or through the mail. Walk-in contacts ma^y entail only a relatively 
few patrons who are either located or visiting in the Center's area or who have 
come for more extensive research of the Center's collection. The latter reason 
may become relevant if the Center develops an extensive curriculum and instructional 
materials collection. 

The content of the Query Service is unpredictable except by extrapolation 
from the experience of other information centers. One of the major reasons for 
proposing these services is to discover what the nature of users* requests may 
be and what the Center's capability to respond will be. Based on the experience 
of several centers, approximately 75% of the requests will be filled with relatively 
simple reference or routine search and retrieval efforts. A significant number 
of queries may be met with standard response "packages" when the Center has been 
in existence long enough to have built a substantial inventory of such "packages." 
Another 20% of requestscan be expected to require more extensive and tostly 
responses because of their complexity, novelty, or demand for information not 
readily retrievable from any accessible source. It is estimated that approximately 
5% of information requests will not be answerable because the question falls 
outside of the legitimate scope of the center's mission or because the information 
requested cannot be located (and may not exist). 

Based on the experience of several ERIC Clearinghouses and consideration of 
upper limits on the estimated sizes of the several target groups (see Planning 
Report* Chapter III), 5,000 requests per year seem to be the outside limit, even 
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for an established two- to three-year old operation. This estimate assumes 
that active users will, on the average, make between two and six requests per 
year (a going rate experienced by several centers) and also that the initial 
response is free (although fees may be charged for more extensive work, such as 
computer searches). 

The test format focused on providing for both telephone hotline and mailed 
requests. The hotline (415/841-2501, call collect) was installed on November 
13th. A toll-free INWATS line (800/227-4535) was in operation between Decenber 
14, 1972, when the Far West Laboratory moved to San Francisco, and March 30, 
1973. Both the hotline and mail service were announced in the Newsletter sent 
to about 150 sponsors, performers, trainers, and R&D employers. 

Pilot test results for the Query Services are quite mixed. On the one 
hand, relatively few persons actually used the services. In spite of announcment 
of the services to almost 200 persons, only nine callers requested formal informa- 
tion services and only four of their requests required extensive computer and 
manual searches. On the other hand, the user survey indicates that the hotline 
received a weightt^d average of 8.20 points, ranking fifth among the 14 alternatives 
and that the mail query received 7.39 points, ranking sixth. Degree trainers are 
consistently high in their priorities, allocating 21.5 of their 100 points to 
these two services. Sponsors, nondegree trainers, researchers, and developers 
follow in that order. Slightly more than half of the respondents in both trainer 
groups and half of the developer respondents reacted positively to the Query 
Service. The latter group was less clear than the trainers about how they v/ould 
make use of the service, with only one specifying that it would be helpful **for 
developing programs--! could find models, consultants and get instant feedback." 
Nondegree trainers stated only that it was useful for "specifics" and for getting 
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information "fast." Degree trainers, on the other hand, had some specific purposes 
in mind--"for developing a research, proposal or a new program," "to answer major 
questions for planning and delivering instruction," to handle "official institu- 
tional inquiries about data, consultants, and planning institutional functions." 
Respondents in all three groups were concerned about the expense and economic 
feasibility of a hotline--indicating that costs would have to be "reasonable." 
A few nondegree trainers indicated a preference for mail over phone inquiries. 
Researchers were divided—two indicating that the services would be useful and 
two indicating they would not be useful. In general, the researchers were doubtful 
that the inquiry service would be able to turn up much that they didn't already 
know, but they might use it as some kind of check. 

When asked specifically about the usefulness of a telephone query service, 
many users indicated either that they av*e .lot oriented toward using a telephone 
service for search purposes or that they prefer to do their own searches. They 
commented: "Most academics would deal in print, unless they want a specific 
title"; "I don*t need information in that much of a hurry"; "No— I would search 
myself"; "What you need you can get without a formal connection"; "Td be prone 
to call a specific person"; "I don't like the phone." 

Interestingly enough, however, there were about 45 other calls logged in 
the record books. Many of these callers used the hotline to inquire about a pre- 
AERA conference on personnel and training. Some were responding to the survey 
of user information needs. Others wanted to contact members of the Far West 
Laboratory staff about field testing or use of specific products. A few were 
curious about progress on the pilot test. 
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The pilot test and user interview data clearly suggest that operation of a 
Query Service is a part-time job. The requests received, however, were demanding 
in the sense that they required a substantial knowledge of the information base 
and how to search it. In a couple of cases, the most useful outcome was 
primarily to indicate that none of several different kinds of searches had produced 
more than tangential retrievals and that there just did not appear to be much 
relevant information on the specific requests. What seems to be indicated is the 
following: (1) Query Services appear to be valued, especially by degree trainers 
and, to a slightly lesser extent, by sponsors and researchers; (2) the services 
would have to be promoted and would probably grow slowly as satisfied users make 
new requests and tell their colleagues; (3) initial query negotiation and prelimi- 
nary searches could be handled by an intelligent and knowledgeable junior professional, 
if easy access to more experienced professionals is provided; (4) the searcher must 
have access to moderately extensive, high-quality collections and,^ if at all possibler 
to an ERIC computer search system [because of their relevant content, NTIS and 
PASAR (Psychological Abstracts Search and Retrieval) services should be used when 
indicated]; and (5) initially, the work load will be low and may never require 
more than a fraction of the time of one competent searcher. Moreover, the search 
work load may be quite variable. Consequently, the search service should be made 
part of a larger operation, either within NIE itself, within one of its Research 
Training contract projects, or an ERIC Clearinghouse. 

Cost Estimates * If the Query Service is incorporated within one of the 
agencies listed above, it would probably represent a relatively small marginal cost. 
If promoted by announcements in Newsletter issues sent to 400-500 persons, the 
number of formal information searches might be estimated between 40 to 100 in the 
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first year of operation, although several hundred simpler requests might be logged. 
Our guess is that, in the first year, between 10% and 20% of the time of one 
information specialist, who is supported by a budget for telephone, computer 
searches, and clerical services, would be required. 

Given the low incidence of calls, a toll-free INWATS line, which could cost 
approximately $4,000 to $5,000 a year, does not seem cost-effective. A "call 
colKect" policy may be justified, however. We estimate that the first year of 
this service would cost less than $6,000, including salaries, benefits, communica- 
ticns, computer searches, and overhead. Note, however, that this does not budget 
for collecting, organizing, or processing information in any form other than 
handling mail and telephone inquiries. It is simply an "add-on" cost. 

C. Summary 

Table 5 below summarizes the "cost preference" data on the estimates presented 

in the previous section. The first three options listed in Table 2 can be combined 

into a single, relatively inexpensive package of related announcement and current 

awareness services. This package accounted for over 40% of the priority point 

11 

allocations made by the pilot test users. The cost of this package is estimated 
at $10,000 to $12,000 for the first year. Second-year costs would probably run 
$12,000 to $15,000, on the assumption that some increase in volume of service would 
occur. Quality service at this cost will be obtained, however, only if the work 
is accomplished as part of a larger, related personnel and training R&D activity. 

All three remaining products emerge as possible contenders for next considera- 
tion. There appearsto be some possibility of producing a companion pair of source 
books, one dealing with the literature and the other with case studies. The price 
for the pair might be between $15,000 and $25,000, if the scope of work required 
is modest. 
O Ti 

ERJC Developers allocated 50 points; degree trainers, 44; researchers, 42; non- 
^^amsim dr^gree trainers, 39; and sponsors, 33. 
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TABLE 5 

COST-PREFERENCE DATA ON PROMISING DISSEMINATION ALTERNATIVES 





Dissemination 
Alternatives 


Weighted 
Priority 


Cost Estimate ; 
(See text for details) 


1 • 


Hotline and Mail Query Service 




^OyUUU T 1 rs c year 


2. 


Newsletter 


14.05 


$3,000 to $4,500 


3. 


Abstracts 


12.15 


$1,000 to $1,500 (if ; 
inserted in Newsletter)- 


4. 


Literature Source Book 


10.41 


$12,000 to $17,000 


5. 


Instructional Materials Catalog 


9.02 


$10,000 to $100,000 


6. 


Case Study Source Book 


5.99 


$3,000 to $15,000 



In our personal view, the potential value of the Instructional Materials 
Catalog has been much underrated, possibly because many respondents may have 
considered it in terms of how often they might consult it or how the informa- 
tion in it might be of immediate use to them. In our view, the true potential 
of such a catalog will be achieved and perceived only if it is developed and 
given appropriate dissemination as a tool for creating awareness and facilitating 
access to new or inaccessible instructional resources in priority training content 
areas. Unfortunately, such a catalog may carry a relatively high price tag. 
A simple compilation of lists of available instructional materials could be developed 
at a much lower cost, but would probably have marginal utility for any of the 
target audiences. 



ERIC 



59 

A brief analysis of costs for the next two priority dissemination alternatives 

--Meeting/Conference Support and Walk-in Query Service--may help to place the 

above conclusions in perspective. 

Support Conference and Meetings . The pilot test involved two conferences: 

12 

a 1973 pre-AERA session and a meeting in Columbus, Ohio, in December 1972. 
This alternative received an average allocation of 5.78 points, or eighth, place, 
with notable endorsement by researchers and trainers. The target user interview 
results were not very positive, either— "sometimes useful /'sometimes not," 
"money spent could be put to better use," "expensive and no payoff," "a waste 
of time," etc. Some users did suggest, however, that focused conferences, 
organized on a regional or problem-centered basis » might be a useful communication 
procedure. These low ratings h'f^re nonetheless surprising, since both the Columbus 
and the 1973 AERA pre-session meetings were well attended and received quite 
favorable evaluations from those participating. Although the user data are 
unimpressive, the results of the two pilot test conferences were quite positive. 
The fact that over 100 persons chose to attend these two conferences on their 
own time and on their own project/agency budgets is impressive evidence of their 
worth. Lf the conference is user-problem- focused and well planned and If users 
are prepared in advance, then conferences can be quite successful. If carefully 
organized, the conference can be a cost effective alternative. Our experience 
with the Columbus and 1973 AERA pre-session conferences indicated that these 
types of meetings can be organized for approximately $1,200 to $1,500 each, if 
users pay their own expenses. 



12 

Further information on these conferences and their evaluation can be found in 
Nancy McCutchan's final report, "Management Development for Internship Training 
Programs in Educational Research, Development, Dissemination, and Evaluation," 
August, 1973. 
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Walk-in Query Service * This would require a center which a person could 
visit, consult with knowledgeable technical personnel, and search an information 
and instructional materials collection. This option received a weighted average 
of 5.76 points, ranking ninth on the list of 14 alternatives. The walk-in center 
could be easily part of the "package" discussed above. A subsidiary collection 
of suitable materials might be priced at $1,000 to $2,000 annually. If supported 
for several years, it could^ with carefui selection, lead to a valuable collection. 
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IV TARGET GROUP PRIORITIES 

The two previous chapters have summarized data obtained from representa- 
tives of several target groups regarding their perceptions of priorities for 
information content, purposes for seeking information, and the kind of 
products and services they most valued. This chapter examines their 
perceptions of the relative priorities this project should observe for 
serving the information needs of the target subgroups. Whom to serve and 
with what priority were early concerns of the project staff. 

It seemed unreasonable that all target groups would be of equal priority* 
Indeed, in preliminary pilot tests, NIE staff had allocated over three-fourths 
of their priority points to only three of the six major target audiences. 
These three groups were sponsors, performers, and trainers. Given this 
information, the project decided to focus further field test data gathering 
on these three major groups; however, because of its large size, as well as 
possible differences between programs, the trainer group was divided into 
two separate samples--degree trainer and nondegree trainer. Similarly, in 
the performer group, researchers and developers were also analyzed separately 
because of possible differences. 

Because NIE personnel expressed concern that their perceptions of target 
group priorities might not be similar to those of other target groups, 
respondents in each of the five groups (sponsors, researchers, developers, 
degree trainers and nondegree trainers) were asked to allocate 100 points to 
show relative priorities for providing eduational RDD&E personnel and training 
information system service to each of sixteen subgroups . The averages for the 
five groups and the overall weighted averages are presented in Table 6. 

ERLC 
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The numbers appearing under the headings of sponsors, researchers, deve- 
lopers, degree trainers, and nondegree trainers represent averages based on 
responses from 4, 4, 8, 7, and 8 persons respectively. The percentage of 
usable returns is good and, although these samples are small, they are repre- 
sentative. The averages suggest there is substantial but not complete agree- 
ment among the priorities of the fl^e groups. This impression was confirmed 
when the averages within each column were rank ordered and rank order corre- 
lations were computed. See Table 7. (Only the 16 specified subcategories 
have been used to compute rank order correlations.) AIT correlations are 
significant at the .05 level. In other words, rank ordered average priorities 
for the five groups agree beyond chance levels. 



TABLE 7 



RANK ORDER CORRELATIONS AMONG FOUR TARGET SAMPLES 
REGARDING RELATIVE PRIORITY ASSIGNED TO SIXTEEN 
SUBGROUPS WHO MIGHT BE SERVED BY AN RDD&E PERSONNEL 
AND TRAINING INFORMATION SERVICE 



GROUP 


Sponsors 


! 1 

: 

Researchers \ 


Developers 


Degree 
Trai ners 


Nondegree 
Trainers 




r 


.77 ; 

1 


.70 


.57 


.51 




.77 


t 
I 


.82 


.84 


.62 




.70 


.82 ,; 




.85 


.74 


Degree Trainers. . . 


.57 


.84 ; 


.85 




.74 


Nondegree Trainers 


.51 


.62 


.74 


•74 

! 

I 





ERIC 
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Sponsors most agree with researchers, then developers, degree trainers 

and finally nondegree trainers. The largest rank order differences between 

sponsors and researchers are found in their relative priorities for "model 

13 

developtsrs," "nondegree instructors," "evaluators ," and "planners"; 
sponsors place greater emphasis than researchers on "model developers" and 
"nondegree instructors" and less emphasis on their own subgroups of "planners" 
and "evaluators." The largest differences between sponsors and developers are 
found in their relative priorities for "planners" and "managers." This may 
reflect a different interpretation of the labels. Developers place a much 
higher priority on "planners" and a much lower priority on "managers" than do 
sponsors. 

Sponsors and degree trainers are most discrepant on priorities for 
"planners" and '^nondegree trainers." Degree trainers, like the researchers 
and developers, give "planners" a high priority, while the sponsor groups 
assign "planners" a middle value. Degree trainers (and nondegree trainersl) 
assign a relatively much lower priority to the "nondegree trainer" group. 
Nondegree trainers, like the developers, attach relatively little value to 
"managers." These major discrepancies between sponsors and nondegree trainers 
(in which nondegree trainers tend to assign much lower priorities to "managers 
and to themselves) account for the relatively low rank order correlation of 
.51. 

The priorities of researchers arelnost like those of developers and 
degree trainers and least like those of nondegree trainers. The relative 
priorities differ most in priorities allocated "managers," "model developers" 

13 

To reduce the confusion between labels for groups assigning and receiving 
priorities, the receiving group label has been placed in quotation marks. 
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Researchers place a much higher priority than developers on serving information 
needs of "managers" and "material developers," while developers place relatively 
greater emphasis on serving needs of "model developers." Researchers differ 
most from degree trainers in placing relatively greater priority on serving 
"materials developers " and lesser emphasis on serving "degree instructors." 
The major differences between researchers and nondegree trainers is in the 
latter group's lack of priority for "managers," and to a lesser extent in 
nondegree •i;rainer's relatively greater priority for "model developers" and 
"preservice students." 

The correlations between the developers and both groups of trainers are 
substantial. Between developers and degree trainers, there are relatively few 
important differences. In terms of rank orders, the largest difference is in 
the relative importance of "state educational agencies"; in terms of point 
averages, it is in the importance of "model developers." Degree trainers 
place relatively less importance on serving "model developers" and relatively 
more importance on serving "state educational agencies" than do developers. 

In terms of rank order differences, developers and nondegree trainers are 
most discrepant in the relative value attached to "nondegree trainers." 
Strangely, developers attach much more importance to serving this group than 
do representatives of this group, f^ndegree trainers (like degree trainers 
attach more importance than developers to serving "state educational agencies." 
In average points, the largest difference between developers and nondegree 
trainers is in the importance of serving "planners." 

Finally, the two trainer groups differ most, in rank orders, on the real- 
tive importance of serving "managers"; in points, on serving "planners" and 
"others." 
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In spite of these various differences, it is important to realize there 
is substantial agreement among these groups. Therefore, it seems reasonable 
to assume that averaging the data for the five groups \\ould lead to a more 
satisfactory estimate of priorities, because it is based on a larger number of 
cases, than would using the data on any one group alone. However, unweighted 
averages were not computed- Instead, the priority data itself was used to 
compute, by iteration, a weight to be assigned to the allocations made by 
each sample group. The procedure has been detailed in a previous report. After 
several iterations, the computed weights stabilized with the following rescaled 
values (to equal 1.000): sponsors .231, researchers .217, developers .195, 
degree trainers .214, and nondegree trainers .143. Developers, researchers, 
and degree trainers are assigned weights very close to those they would have 
if all groups were treated equally (.200); sponsors receive a relatively higher 
weight and nondegree trainers, a much lower weight. These weights are used in 
computing the weighted averages reported in the tables in Chapters II and III. 

With this explanation of how weighted priorities were derived, the reader 
should return to Table 6 and examine the last column, at right which 
indicates the weighted priorities for the 17 subgroups, and shows the sum of 
weighted priorities for subgroups comprising the group. 

A note of caution : If students, employers, and practitioners had been 
included as sampled groups in the study, the final set of weights would be 
different, probably attaching more importance to serving such groups. The 
differences might not be great. Note, for instance, that the nondegree 
trainers assigned a lower weight to serving their own group than did sponsors, 
developers, or degree trainers. However, as we scan from left to right across 
Table 6, from sponsor to nondegree trainer, a distinct trend in assignment of 
priorities to n:^oups other than sponsors, performers, and trainers can be 
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observed. These are the total priorities assigned to these other groups: for 
sponsors, 22.5%; for researchers, 18.2%; for developers, 23.2%; for degree 
trainers, 29.2%; and for nondegree trainers, 38.0%. The differences are small, 
but there is a suggestion trainers are more "practice-oriented" than sponsors, 
researchers, or developers. 

Related evidence can be found in Table 7. The rank order correlations in 
this table display a simplex pattern in which the highest correlations are 
found closest to the diagonal. This kind of pattern occurs when variables 
are arranged so that those most highly correlated with each other are placed 
next to each other in the matrix and those with lower correlations are placed 
correspondingly further away (e.g., sponsors correlate least with, and are 
farthest apart from, nondegree trainers in their row and column arrangenient) . 
The point is that Table 7 was not artificially rearranged to create this pattern. 
Rather it arises from the order in which the target groups were listed.* It is 
our hypothesis that, if data were collected from all possible groups, a 
simplex pattern of correlations could be created and it might require little 
or no reordering of groups on Table 6. Stated another way, we are guessing 
that the perceptions of priorities of employers, for instance, would be more 
like those of the students and practitioners, to whom they are adjacenc on 
the list in Table 6, and least like those of the sponsors and performers, from 
whom they are the most distant on the listing. This guesswork suggests that, 
vf NIE is satisfied that sponsors, performers, and trainers are the major 
groups to be served (the data in Table 6 indicate this), then the present data 
will be quite useful in providing guidance for the design of an information 
system. However, if, for some reason, much greater importance should be 
attached to providing, say primarily emplo^iirs and practitioners with personnel 
and training information, then the results of this particular user study might 

® order of listing was based on an a priori model of similarity of information need. 
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be misleading. 

Summary analysis . Five different groups were sampled. Representatives 
of each target group were asked for their perceptions of the priority that 
should be attached to serving the possibly different needs of various groups. 
The five groups were generally in agreement regarding priorities, although they 
displayed some difference. Overall, they assigned three-fourths of their priority 
points to the three major target groups: sponsors, performers, and trainers. 
The students, ©nployer, and practitioners were not totally disregarded but their 
perceived significance was much smaller. Performers, sponsors, and trainers 
received approximately equal priority and overall weighted priorities for sub- 
groups within these groups do not differ greatly. Table 8 displays the subgroups 
rank ordered for overall weighted priorities. 

TABLE 8 

TARGET SUBGROUPS RANK ORDERED BY OVERALL WEIGHTED PRIORITY 





Subgroups 


Priority Weighted | 


1. 




10.49 


2. 


Degree Program Directors 


9.80 


3. 




9.74 


4. 




8.91 


5. 




8.71 


6. 




7.39 


7. 




6.95 


8. 




6.47 


9. 




6.17 


10. 




4.71 


11. 




4.62 


12. 


Local Educational Agencies 


3.83 


13. 


Inservice (on-the-job) students 


3.76 


14. 




2.89 


15. 




2.77 


16. 




2.16 


17. 




0.61 


Total 


99.98 



o 

ERIC 
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V. AN RDD&E PERSONNEL AND TRAINING COLLECTION 
A. Introduction 

The two previous chapters summarize project findings regarding users' 
needs. This chapter focuses on the organization and content of the informa- 
tion that may be provided. There are two major sections. The first 
describes project experience with indexing and classifying an experimental 
educational RDD&E personnel and training collection. The second section 
summarizes findings, reported at length in another document, regarding the 
quantity and character of RDD&E content. 

Chapter IV of the Report of the Planning Phase had outlined an informa- 
tion system model which dealt with nodes (originators, processors, and users) 
functions (collection, processing, etc.) and information flow 
(characterized by content, subject orientation, format, levels of formaliza- 
tion, accessibility, etc.). The relative advantages and disadvantages of 
informal, semiformal, and formal communication were examined. The formats 
of RDD&E Information were located in a matrix based on their classification 
by structural form and by levels of formalization. Finally a preliminary 
scheme for classification by Information content and subject orientation 
was outlined. 

B. Indexing and Classification Experiment 
Over 1,000 items were collected and classified by the project. After 
reviewing alternatives, the project recommended that NIE consider an "ERIC- 
like" system in which surrogates, including both citations and abstracts 
are created. These should be deeply indexed (I.e. with 8 to 12 terms) by 
standard ERIC thesaurus terms so the collection would be fully compatible with 
the ERIC collection. Search of such a system would thus be quite similar 

ERIC 
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to a manual search of ERIC. One would first look up the ERIC term in an 

index and locate the document number, then locate the citation and abstract 

in a looseleaf notebook, decide if the item seemed relevant, and if so, 

locate the physical item. This idea was tested by creating a collection 

of over 200 citations and abstracts of significant RDD&E literature which 

were indexed and organized as a looseleaf catalogue of annotated literature. 

We found it advantageous to use a looseleaf format with only one item to 

a page, and to reproduce several copies. This enabled us to maintain one 

serial order copy, which could be used when searching by thesaurus terms, 

while arranging other copies to facilitate browsing by subject interest 

(e.g., if an item dealt with research and evaluation, the abstract could 

be placed with the research abstracts and also with the evaluation abstracts). 

These browsable collections of abstracts are recommended as one po5»sible way 

to increase access to, and retrieval of, the most significant RDD&E literature. 

The cost of deep indexing and abstract preparation (if abstracts cannot be 

found in ERIC or other sources) is substantial and seems warranted only if a small 

collection of high quality documents is desired. Note, however, that lesser 

quality documents can be handled by entering only the citation and indexing 

under only a few key terms. As a byproduct of this experiment, the project 

published Educational RDD&E Personnel and Training Abstracts , a looseleaf 

collection of over 200 abstracts of significant RDD&E literature. 

After considering several arrangements for classifying and physically 
storing items, we opted for a "browsable" collection with items organized 
by the classification scheme reported in Appendix C of the Report of the 
Planning Phase (Hood, McCutchan, Aldrich, and Wolf, 1972). This scheme 
recognized four major categories: (1) books, papers, and reports; (2) 
^ instructional materials; (3) personnel supply and demand information, and 

ERIC 



71 



(4) other personnel and training information (e.g., directories, program 
descriptions, etc.). A fifth category consists of bibliographic references, 
ERIC, and the serials collection. Materials are arranged within these 
categories by subject matter headings (e.g., methodology, theoretical 
and conceptual issues) or information types (e.g., case studies, biblio- 
graphies, assessment instruments). The classification scheme is not 
technically elegant, and is probably inconsistent with any major classifica- 
tion system. But, given the very important assumptions that the collection 
was to serve as a reference core for an information center rather than 

as an archival library and that only a few thousand items would be main- 
tained, the major virtue of the proposed approach is its practicality. 
Our pilot test searches were usually confined to one or at most two 
categories. Within these major categories, the number of items under a 
subject heading (e.g., "Research Methodology" or "Policy, Roles and Strategies- 
Diffusion) proved to be so small that browsing or searching all items, 
or reshelving items was not difficult. It was also easy to instruct 
visitors on how the collection was organized so they could search or browse 
for themselves. To some extent, mainly because of the small size of the collection, 
we have taken an experimental attitude toward the classification problem. 
Our view is that materials need to be arranged in ways that will facilitate 
retrieval of information (or instructional resources). When one is looking 
for information, as opposed to looking for specific documents, it helps to 
find items with similar content located together. Obviously, items may be 
groupo'd in many ways and classified under several possible headings. We 
tended to resolve these dilemmas by putting items where we thought they 
would have most value if a person were searching under that 
heading. The problem of multiple classification can be handled by 
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arranging multiple copies of the Abstracts into browsable folders by subject 
headings. If these folders are shelved under their headings, they provide 
the browser with a quick overview of (a) what is on the shelf, (b) what is 
on shelves elsewhere, or (c) what should be on the shelves but may be in 
circulation. 

At the termination of the project, we are still experimenting with ideas 
for subject headings. We have learned that one must work with the 
collection for some time to find the groupings that best facilitate search 
and retrieval of the kinds of information that are most frequently sought. 
Fortunately, the experimental collection was sufficiently small that minor 
refinement or even major rearrangement is not an arduous task. Possibly the 
most tedious part is simply revising the "call numbers" in the abstract 
notebooks. We seriously doubt that our experimental classification system 
would accommodate a collection much larger than a few thousand items. It 
obviously assumes that the collection is not to be merged with a larger 
library collection. It places a premium on physical browsing and search 

for information or content within documents or other materials rather than 
search for the document or the materials per se . 

We have encountered the need for some auxiliary index terms to provide 
differentiations not provided by the ERIC thesaurus. This lack can be a probl 
in a content area such as "development." But auxiliary terms or codes are 
needed especially for instructional materials to indi^jate more accurately 
the medium*, relevant target audiences, level of difficulty, etc. Currently 
the problem is not serious, since the number of instructional materials other 
than books within any one subject category is not too large to permit direct 
inspection. Auxiliary terms or codes would save tnis extra work which all 

*Several media classification systems exist. We believe that Umy-Rousseau*s 
(1972) may be one of the best in terms of complete specification for nonbook 
materials. 
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too frequently leads to the finding that the available materials are inappro- 
priate. We note that the published Abstracts list only the standard ERIC terms. 

C. The Quantitative Content of RDD&E 
literature and Instructional Materials 
This section summarizes infomation which has been provided in a separate 
report titled The Domain of R&D Training Resources . The project's experimental 
collection was coriiidered too small to provide reliable estimates of the total 
content of literature and other materials that would conceivably be germane to 
users' needs. Several larger information bases including ERIC, Subject Guide 
to Books in Print , and three instructional materials collections were examined 
to develop more reliable estimates of the quantities of materials available 
within subject content categories. 

Examination of these data bases and review of the literature on documentation 
suggests that ninety percent of the educational R&D resources are largely informal 
and fugitive. Perhaps ten percent reach a semi formal level where they may be 
in published form and may be referenced. Possibly only one or two percent achieve 
"hard-cover" commercial publication and much less than one percent are subjected 
to the test of the R&D development cycle. Probably over 95 percent of all instruc- 
tional materials is in printed form. The number of fully developed and tested 
materials is probably much less than 100 items; perhaps 1,000 have undergone 
minimal development and may be generally accessible although not necessarily 
easily so. And there are possibly 10,000 books and textbooks which bear on R&D 
training content as defined by the project. ERIC documentation suggests there are 
more than 40,000 relatively recent (1966-1972) journal articles and reports which 
provide the knowledge base for R&D training content. The total relevant published 
literature my be closer to 100,000 items. 

There are significant but far from perfect content correlations among the 

ERIC 
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three data bases. ERIC descriptor counts prove to be better predictors of 
counts of nonbook instructional materials subject categories than do count of 
Books in Print . Almost all the instructional materials are located in class- 
ical "educational research categories," with only three of eleven major content 
categories containing over half of the books and textbooks and over forty 
percent of the nonbook instructional material s^ By contrast only four pe»"cent 
of all instructional materials were classified in the two general content cate- 
gories of (a) Design and Development and (b) Evaluation. However, examination 
of trends in ERIC as well as other evidence, suggests that the rate of growth 
may be greater in these less well-established but high interest areas. 

Table 9 reports comparison counts in three categories: (1) nonbook 
instructional materials, (2) books and monographs, and (3) ERIC journal articles 
and other ERIC documents. The ERIC counts are estimates based on counts of 
ERIC descriptor usage. Books and nonbook counts are counts of actual titles. 
All counts are considered to be conservative underestimates. The Domain of 
R&D Training Resoruces should be consulted for definitions of subject content 
categories, mehtods of making estimates, etc. The content counts are useful 
in indicating the relative quantities of materials available in different sub- 
ject content categories. 



ERIC 
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TABLE 9 

COMPARISON OF CONTENT COUNTS IN 
THREE INFORMATION BASES 







Instructional Materials 


ERIC Journals & 




Subject Content Category 


Nonbooks 


Books 1 


Other Documents 








Per- 




Per- 




Per- 






No. 


cent 


No. 


cent 


No. 


cent 


1 - 


Policy, Planning, Change, Innovation 


79 


10.7 


738 


10.7 


3280 


8.2 


2. 


Research 


71 


9.6 


791 


11.5 


3920 


9.8 


3. 


Design and Development 


44 


6,0 


82 


1.2 


2800 


7.0 


4. 


Dissemination and Comnuni cation 


29 


3.9 


316 


4.6 


920 


2.3 


5. 


Evaluation 


40 


5.4 


109 


1.6 


3240 


8.1 


Subtotal 


263 


35.6 


2036 


29.6 


14160 


35.4 


6. 


Science and Technology 


5 


0.7 


49 


0.7 


600 


1.5 


7. 


Data Analysis, Statistics, Other 


139 


18.9 


1127 


16.4 


2440 


6.1 




Analyses 


62 


8.4 


147 


2.1 


SOO 


2.0 


8. 


Objectives and Criteria 


9. 


Tests, Measurement, Data Collection 


55 


8.8 


617 


9.0 


5760 


14.4 




Instruments 














10. 


Methodology, Technique, Theory 


79 


10.7 


1406 


20.4 


6560 


16.4 


Subtotal 


350 


47.5 


3346 


48.6 


16160 


40.4 


11. 


Disciplines and Specialties 


11 


1.5 


614 


8.9 


520 


1.3 


12. 


Skills, Training, Professional 


28 


3.8 


169 


2.5 


2480 


6.2 




Education 














13. 


Personnel 


5 


0.7 


280 


4.1 


2240 


5.6 


14. 


Content and Media 


80 


10.9 


435 


6.3 


4400 


11.0 


Subtotal 


124 


16.9 


1498 


21.8 


9640 


24.1 


Total 


737 


100.0 


6880 


100.0 


139960 


99.9 
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Table 10 below displays only the instructional materials data, reorganized 
tn descending rank order by books counts, and excluding three categories listed 
in the previous table which pertains primarily to personnel and training infor- 
mation ger S£ rather than to instructional content. 



TABLE 10 

COUNTS FOR ELEVEN R&D SUBJECT MATTER CONTENT CATEGORIES 
FOR BOOK AND NONBOOK INSTRUCTIONAL MATERIALS 



Content Category 




Nonbooks 


Books 




Count 


Per- 
cent 


Count 


Per- 
cent 


Methodology, Technique, Theory 


79 


11.4 


1406 


24.2 


Data Analysis, Statistics 


, Other Analyses. 


139 


20.0 


1127 


19.4 


Research 


71 


10.2 


791 


13.6 


Policy, Planning, Change. 


Innovation . . . 


79 


11.4 


738 


12.7 


Tests, Measurement, Data 


Collection Instru. 


65 


9.4 


617 


10.6 


Content and Media . . . 




80 


11.5 


437 


7.5 


Dissemination and Communi 


cation 


29 


4.2 


316 


5.4 


Objectives and Criteria 




62 


8.9 


147 


2.5 


Evaluation 




40 


5.8 


109 


1.9 


Design and Development 




44 


6.3 


82 


1.4 


Science and Techno! oav . 




5 


0.7 


49 


0.8 


Totals 


693 


99.8 


5819 


100.0 



This table affords a rough estimate of the relative quantities of materials 
available. The rank order of counts for books and nonbook materials are quite 
similar (the rank order correlation is .81). The top five categories account 
for 80 percent of the books and 62 percent of the nonbook instructional 
traterials. These categories are well -establ i shed, traditional, educational 
research areas. By contrast the three "DD&E" areas ("dissemination and com- 
munication," "design and development," and "evaluetion" account for less than 
^ nine percent of the books and only sixteen percent of the nonbook materials. 
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To obtain an estimate of the formats for nonbook materials, we classified 
317 "RDD&E relevant" items found in Resources for Performance -Based Educati on 
(Houston, et al.. 1973). These results are reported in Table 11. 

TABLE 11 

FORMATS OF R&D RELEVANT TITLES IN 
PERFORMANCE- BASED EDUCATION 



Format 


No. 


Per- 
cent 




134 


-♦2.3 




59 


18.6 




47 


14.8 


Slides, tapes and films trips .... 


44 


13.9 




12 


3.8 




8 


2.5 




7 


2.2 




G 


1.9 


Total 


317 


100.0 



These studies of content indicate there is a very large base of publishrl 
literature which constitutes the foundation for the content of educational 
RDD&E. Well-established content areas contain relatively large numbers of 
books, but newer areas such as "evaluation" or "product development methodology" 
are much less well represented, either in journal or book form. Instructional 
materials, other than books, are relatively hard to find. Our searchers suggest 
that fewer than 1.000 truly useful items may exist. Given the differences in 
the estimated sizes of these resource population (i .e. , approxiamtely 1,000 
nonbook instructional materials, 10.000 books, and 100,000 journal articles 
and reports) substantially different strategies are required if a user-oriented 
collection is desired. This problem is treated in the next chapter. 
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VI, Evaluation of RDD&E Materials 
A. Introduction 

Initially, the project aimed to investigate alternatives and make 
recommendations for the evaluation of information and materials to be included 
in the proposed personnel and training information system. Appendix D of the 
Report of the Planning Phase (September, 1972) briefly explained four types 
of evaluation: (1) selection of items for input, (2) evaluation of materials 
to determine the appropriate category for analysis and category for dissemina- 
tion, (3) day-to-day formative and quality control evaluation of services and 
products, and (4) overall evaluation of the system through preestablished 
feedback mechanisms. 

Later in the project, the interest of the sponsor shifted primarily to the 
evaluation of instructional materials. This subject was examined in depth and a 
report was prepared titled Evaluation of Instructional Materials for Educational 
RDD&E Content Areas (Hood, July 1973). The content of that report is summarized 
in the next section. 

B. Evaluation of Instructional Materials 
At least six different purposes for evc'^uating instructional materials were 
identified, namely: (1) selecting items for accession, (2) selecting items for 
indexing and abstracting, (3) purging obsolete, low value and duplicative items, 
(4) selecting and critically evaluating items for selective dissemination, (5) 
selecting, evaluating, and analyzing items for an instructional materials catalogue, 
and (6) selecting and evaluating items for intensive dissemination/promotion. 
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Acquisition, Indexing, and Purging 

The first three evaluation purposes, initial selection for acquisition, 
evaluation for retention, and purging are directly related. They have the 
general goal of controlling the cost and size of a collection and of 
maintaining the quality of its information. With the exception of the case 
of a comprehensive instructional materials center, whose mission would be to 
compile an exhaustive collection of all instructional materials in educational 
RDD&E or related areas, any other collection will probably be selective with 
its acquisitions and guided by the requirements of its actual or potential 
user populations and by the limitations of its budget. 

Acquisition Policy . As this design project has noted in Chapter II, the 
needs and requirements of various RDD&E personnel and training 
user populations are quite diverse. Defining priorities with respect 
to user populations and their needs is essential in order to provide 
general guidance; but, in practice, the ultimate responsibility for 
selecting and acquiring materials will rest on the judgments of one or 
a few professionals. This judgment can and should be guided by an 
explicit acquisition policy. Issues to be considered in developing such 
a policy include the following: 

° What is the primary purpose of the instructional materials collection? 

^ Is it to be comprehensive or selective? 

^ Will it reflect more what is available (archival -oriented) 
or what users need (service-oriented)? 

^ Will it focus on readily available materials, fugitive 
materials or both? 

° Will it attempt to include incomplete materials, e.g., 
prototypes still under development? 
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^ Win it have priorities with respect to specific content 

emphases (e.g., actively searching for materials in 

designated priority areas)? 
^ Will it have special media priorities (e.g., attempt to 

build relatively large audiovisual or game-and-simulation 

collections)? 

To what extent will self-instructional or other "learner 
active" materials be sought in preference to the more 
conventional published textbooks? 

To what extent will the collection be open to general staff 
or outside visitor use (accessibility of the collection and 
relates to wear and tear, loss, and the frequency of reshelving)? 
^ To what extent will circulation or interlibrary loan be 
permitted? 

^ How will the collection be related to other existing collections, 
say NIE library or a personnel and training document collection 
(such ties will entail a need to duplicate copies, to correlate 
indexing, etc.)? 

Which training needs have what priority? ror example do undergraduate 
needs have priority over the needs of graduates. And which of these 
needs shall be given priority, preservice vs. inservice; part-time 
vs. full-time R&D practitioners; established, high-demand content areas 
(e.g., research methodology) vs. emergent, low-demand content 
areas (e.g., formative evaluation, R&D policy formulation)? 
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° To what extent will an attempt be made to make the collection 
proactive (i.e., able to anticipate future needs and represent the 
best or possibly all that is available in areas where there is only a 
predicted as opposed to an empirically established user need)? 
° To what extent will the collection be delimited to core "disci- 
plined inquiry" vs. peripheral content areas (e.g., where does 
educational RDD&E content merge over into other educational 
professional training content or noneducational R&D training content)? 
° To what extent will the collection focus on quality vs. quantity, 
i.e., having only the best as opposed to having everything available 
in any specified content area? 
^ To what extent will older materials or earlier versions of 

materials be retired in favor of new or revised versions? 
° What emphasis will be placed on retaining materials for historical value? 
Once questions such as the above have been answered and an explicit 
acquisition policy has been formulated, actual acquisition of instructional 
materials may be started with at least some assurance that the collection 
will be established and maintained according to a systematic plan. 

Indexing . Because we anticipate that not every item acquired 
will contain relevant content or be of adequate quality, instructional materials 
acquired should be fully processed and indexed only if they constitute a 
valuable contribution to the collection and its potential for meeting users' 
needs. The aim in screening a fter accession is to reduce processing and 
collection maintenance costs, but more importantly to eliminate poor and 
inappropriate items. 



ERIC 



Purging . Purging instructional materials collection of outdated, 
inefficient, and irrelevant materials is possibly one of the most needed, 
but least well -performed functions for most instructional materials 
collections. Unless the collection acquisition policy places great 
emphasis on its archival function (usually at a risk of not serving 
current users* needs well) there will be a need to periodically review 
the collection and purge it of its less valuable items. 

Whether considering an item for accession, for indexing or ab- 
stracting, or for discard, essentially the same judgmental process and 
criteria are required. The basic question is how does placing (or 
removing) this item affect the overall value of the collection and the 
costs of maintaining it? 
Recommendations 

The need for high standards in selecting instructional materials is 
not fundamentally different from a similar need during documentary accession. 
The creation and maintenance of a collection of the best instructional materials 
available in each pertinent subject field should be a basic objective. 
This selection should be done systematically, keeping in mind the broad 
purposes of disciplined inquiry in education as well as the specific 
interest and current priorities of particular user groups. 

Possibly the best advice we can give with respect to criteria for 
developing and maintaining a collection is that, at least initially, the 
collection should aim at quality rather than comprehensiveness and, within 
each subject category, it should attempt to secure the best materials obtainable. 
The acquisition policy should be proactive and it should have priorities. 
Search and acquisition should be especially active in areas where critical 
needs exist or can be projected. This may mean setting lower quality standards 




se priority areas and in making special efforts to search out materials 
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that are still in drvelopment. For example, if continuing education is 
considered more important- than preservice education, then emphasis should be 
placed on acquisition of self-instructional materials 

In short, we recommend that the collection be responsive to user needs and 
policy requirements. It should not be merely a passive reflection of what is 
conveniently available* Our statistical estimates of quantities of 
instructional materials accessible in different content areas indicate that if 
the latter policy is followed, the great bulk of the collection will deal 
with research, experimental design, statistics, tests and measures and oth^r * 
well-established content areas. Only a small part of the collection would 
deal with policy, evaluation, development, dissemination or similar content 
areas* A policy of priorities seems justified at least until one can be 
satisfied that reasonably adequate subcoll^ctions exist for all 
priority areas. When this is accomplished, and if the budget and 
resources are .available, then larger subcol lections of the "easy-to-collect" 
content areas might be undertaken. We see little gain in having a very large 
collection that is representative of what is available if such a collection 
does not effectively meet current and projected user and sponsor needs. 
Evaluation of Insturctional Materials for Dissemination 

This part deals directly with evaluation problems, issues, and recommenda- 
tions as they relate to the creation of an instructional materials dissemination 
effort. The first section of this part (ieals with the overall purpose of such 
an effort, the problems of selecting a set of items and the relevant "costs" 
and "utilities". The second section discusses four evaluation approaches which, 
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because of their different purposes, require different decision models. 
The final section contains practical recommendations for the alternative 
evaluation approaches. 

1. Costs and Utilities in Creating a List of Selected Instructional Materials 
At least two significantly different evaluation problems can occur in the 
creation of an instructional materials catalogue or other dissemination effort. 
The more obvious concern is how to evaluate each item of instructional 
material. In our view, the less obvious but more critical evaluation 
problem concerns how to evaluate the alternative forms and uses of the total 
effort (catalogue, list of recommended materials* etc.) These two kinds of 
evaluation problems interact, but it seems wiser to consider the total 
effort problem before considering the problem of evaluating specific, 
individual instructional materials. 

Whether selecting, analyzing, or critically evaluating items for 
dissemination (current awareness), for a cataiociie, or for intensive 
dissemination, presumably the basic aim is to create awareness of, and access 
to, some set of instructional materials. Crucial to the decisions regarding 
dissemination are considerations of (1) objectives of a dissemination effort, 
(2) costs to disseminate and (3) utility to the users. There will be at least 
the following costs to consider regarding s pecific instructional items . 

K Cost to search and acquire information about the instructional materials. 

2. Cost to create and check accuracy of citation. 

3. Cost to analyze the instructional material and obtain information 
about its development, field testing, etc. 
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4. Cost to evaluate the instructional material. 

5. Cost to list and describe the instructional material. 
(Reports outcome of 2,3, and 4.) 

6. Cost to classify or index the instructional material (if a list 
or catalogue is anticipated). 

7. Cost to organize, edit, publish, and distribute the list or catalogue. 

8. Cost to user to gain access to the list or catalogue. 

9. Cost to user to "use" (search out and interpret v^hat is reported). 
10. Cost to user if the listing misleads him in purchasing or using 

less adequate materials or in not purchasing or using what would 

have been more adequate materials. 
Costs 9 and 10 are generally inversely related to cost 3.4, and 5 
especially, but may also be inversely related to 1, 2, and 6. There is definitely 
a tradeoff between cost to producer and cost to user. Moreover, there is 
usually a tradeoff between quantity and quality. As the number of in- 
structional resource items are increased, given a fixed "producer's" budget, 
the amount and quality of information available on each specific item will 
usually diminish. Analysis and evaluation, and the acquisition of sufficient 
information to accomplish either, can be especially costly. For the same 
budget, one can produce a catalogue that only lists and possibly indexes 
citations to a very large .number of instructional materials, or 
conversely, provides citation, analysis, and extensive evaluation of a 
relatively few instructional materials. But which type of catalogue will 
be more useful? 
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Several kinds of utility may be worth considering; the primary one 
usually will be the utility for the user who is searching for instructional 
materials that can be used in training. The users may be the 
target groups we have identified in this project as degree trainers and 
nondegree trainers or the users might even be students themselves, 
especially if the individuals are looking for self-instructional aids to 
acquire new skills or upgrade existing :ikills. Utility for these groups 
may best be considered in terms of the difference that access- to informa- 
tion about instructional materials might make compared to what the groups 
would do without the Information. In other words, would use of a selective 
dissemination, service or an instructional materials catalogue give the 
trainer or student access to better materials than would otherwise be 
available without the service? Would it reduce their cost and 
effort in searching for better materials, or possibly even result in 
their acquiring and using materials in content areas which otherwise 
would not be accessible at all? This kind of differences implies 
marginal utilities which, though hard to measure, are nevertheless meaningful. 

There are other types of utilities which may also be important. For 
instance, the sponsor or developer target groups may be concerned with 
obtaining a general overview of "what is available*' or "the best that is 
available" in particular areas. They may be interested ?n whether or not 
sufficient justification exists to develop particular types of instruc- 
tional materials, keeping questions of possible redundancy well in mind, 
and searching for good ideas for formats and methods, etc. In such cases, 
relatively comprehensive (but hopeful ly well classified and indexed) listings 
may be of greater value than smaller but more selective listings. 



ERIC 



87 



since the more comprehensive listing can represent "sampling frames" from 
which samples may be drawn for more extensive surveys and analysis. 
Because these kinds of needs can be so specialized, it is doubtful 
that any particular format for a catalogue or other listing would be 
completely satisfying. Obviously, comprehensive and accurate indexing 
and descriptions would make such a catalogue more useful as a starting point. 
But the amount of savings in such specialized uses will probably be small. 
It would probably be of greater importance that the list be well documented, 
reasonably recent and inexpensive to update. 
2, Evaluation Approaches 

Too often a particular method of evaluation is selected without due 
regard for the larger measurement and decision assumptions which it may 
implicitly entail. So before considering any specific methods or techniques, 
we shall undertake a broader examination of some of the issues related to 
the evaluation of sets of instructional materials where the intent is 
to convey information about them to sponsors or users. 

Although many dissemination requirements can be described, 
there seems to be a smaller number of evaluation approaches. Possibly, the 
simplest way to group these approaches is by their intent: (1) to select, 
(2) to classify, (3) to criticize or (4) to estimate worth for particular uses. 

Selection appears to be the most fundamental problem since it must 
be a part of any of the other three approaches. These are some of the factors 
bearing on selection:, (a) the number of items to be considered, (b) 
the cost of considering each item, (c) the final number to be selected. 
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(d) the finality of the decision, (e) the risks involved in misassignments 
and (f) the technical characteristics of test and decision processes 
available (accuracy, precision, reliability, validity, etc.)* 

Factors (a) and (b) interact in the practical situation, since the 
total amount of resources (time, money, patience of judges, etc.) available 
for selection will be limited. A large number of items to be considered 
implies a smaller average amount available per item. The final number 
to be selected is also important. The ratio of (c) and (a) establishes 
the selection ratio. The selection ratio has an important bearing on 
the utility of any selection effort (Cronback and Glesser, 1965). 
Whenever extreme selection ratios are encountered (i.e. when most but 
not all are to be selected or when only a few of many are to be selected) 
it almost always pays to employ a sequential selection procedure. In 
fact, sequential selection is usually employed when only a few of many 
items are to be selected. It is less often used, but should also be 
employed, when one is rejecting only a few items. 

Sequential selection procedures relate to factor (d), the finality 
of the decision. An important question to ask is whether the evaluation 
method calls for an irrevocable comrr.itment to accept or reject or whether 
the decision can be reviewed later. Typically there are two classes of rejects: 
those rejected completely, and those rejected for now but held for further 
consideration. Similarly we often find two cl.?*sses of accepted Items: those 
accepted without, qual if ications, and those accepted with qualifications. In 
some selection situations an entire group of rejects may be eligible for con- 
sideration at a later date, but in most cases, a complete reject has no further 
chance. This would certainly be so for a particular is'sue of an instructional 
materials catalogue. By contrast, the ETS selection of products for focused 
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dissemination (Epstein, Margosches, Schrader and Walton, 1971) has given rejects 
a second chance, and has, in effect, placed products in all four classes des- 
cribed above. Cronback and Glasser (op. cit., Chapter 6, Appendix 4 and Appendix 
5) provide a useful technical discussion of how sequential testing improves 
the efficiency of the selection process. 

Undoubtedly the most important aspect of any selection process is the 
consideration of the risks involved in misselection, that is, considering 
the gains =.nd losses attendant on rejecting a "good" or accepting a "defective" 
one. In the dissemination of instructional materials, this usually is the 
losses to the user, the distributor, and the developer of failing to list a 
(rejected) instructional material. The acceptance of a defective item will 
also represent a loss to someone who acquires and uses it, and will eventually 
reflect on those who sponsor, develop, or disseminate the catalogue, list, or 
other media including the instructional materials. 

Consideration of selection risks helps focus attention on the "welfare 
economics" of any educational materials selection problem--the two ways of 
being in error, the number of times errors occur, and the risks associated 
with those errors. A comprehensive concept is the "utility" of the selection 
process itself. The overall utility of the decision process applies to a 
larger number of elements including at least: the number of instructional 
materials to be considered, the selection strategies available, the information 
available, the number of outcome categories, the value of each outcome category 
the cost of gathering information, and the cost of making and implementing 
decisions. 

If decisions can be empirically validated, then there is ample empirical 
evidence that "statistically" established weights will almost always prove 
superior to /'clinically" established weights or other decision rules. A severe 
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problem for evaluation of educational products is that adequate statistical 
validation procedures are virtually nonexistent. We lack accepted criteria, 
means of measurement, and sufficient quantities of data on a broad enough 
range of products to even begin to deal with validity. Indeed, we have only 
the crudest information on reliability. Typically the merit, whether explicit 
or implicit, is at best ordinal, that is a rank ordering in terms of overall 
merit. Until greater attention is given to measurement, especially criterion 
measurement, and to evaluation of utility for those criteria, there will be 
little or no rigorous technical basis for selection. The validity of nearly 
every educational product evaluation procedure is assumed, A few studies 
cite interrater reliability coefficients; but validity coefficients are almost 
nonexistent. 

Lacking this kind of information, most evaluation decisions in the near 
future can at best adopt a strongly Bayesian perspective in which well-informed 
prior distributions (regarding the distribution of estimated utilities) can 
at least be subjected to some empirical test, so that more effective post- 
posterior distributions can be developed. (Aitchison, 1970; Ferguson, 1967; 
Raiffa and Schlaifer, 1961; Schmitt, 1969). Currently our approaches, whether 
analytic or wholistic, remain virtually unexamined and certianly unvalidated. 

Classification . The evaluation problems of classification are more complex 
than those of selection. Basically classification involves the assignment of an 
object to one of a set of categories. The simplest type of classification 
problem uses univariate information. For instance, in terms of some overall 
measure of the worth of instructional materials, they may be classified as: 
not acceptable, poor, good, excellent. For an instructional materials catalogue, 
the classes might be the following: do not list, list only, list with d'r-.s- 
cription, or list with description and recommendation. (By contrast, selection 
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would involve only the decision to list or not list.) The situation in which 
there are several types of information that cannot be reduced to a single 
composite measure is far more difficult. Horst (1954, 1956), Brogden (1951, 
1959) and Magwire (1953) have done much to develop the statistical foundations 
for this type of problem. Unfortunately, these treatments offer little 
practical help in the absence of information which relates evaluation measures 
to payoff measures. But they do provide assistance in conceptualizing the 
problem and in suggesting how one might proceed if adequate data were available. 

Criticism . Unlike selection and classification, which involve well- 
defined decisions leading to relatively precise dispositions, criticism 
is a much less precise activity. However, it can be reduced to a basic 
multiple dimension classification model in that the critic analyzes an 
instructional material , identifies its salient attributes, and then 
evaluates the strengths and weaknesses or otherwise the worth of those 
attributes. Usually an overall appraisal of worth for one or more implicit 
or explicit uses is also attempted. Certainly few critics would recognize 
or agree that their efforts reduce to a classification problem, but we 
believe this is the case. 

We are concerned basically with criticism applied to a set of products, 
not a single product. The problem is usually not to criticize a single pro- 
duct from a particular perspective, perhaps regarding a very specific use; 
but rather to criticize each of a number of alternative products from a 
more general perspective. 

A significant quandary for critics and those selecting or classifying 
is the determination of how well-defined and how valid are the critic's re- 
quirements. Most criticism is based on assumptions that criteria such as 
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authenticity, appropriateness, scope, interest, technical characteristics, 
etc. will be related to ^'ecisions of worth or utility. In most cases these 
attributes appear to be quite logical and enduring. 

An interesting problem is how does the decision maker--whether another 
person acting on the critic's information, or the critic himself--arnve at an 
overall judgment? Sometimes, the critic leaves this problem to another person 
by reporting sufficiently on his analysis and evaluation, to permit another 
person to make his own decision. Indeed, given the variety of settings, 
conditions, and special requirements which different users may encounter, 
well -documented , evaluated criticism, will usually be preferred over a pre- 
cisely stated, but less informative overall grade or set of scores or symbols. 

Evaluation of worth for particular uses . Ultimately, decisions regarding 
a limited set of alternatives with regard to highly specific situations will 
be made by the decision maker(s). With respect to educational RDD&E instruc- 
tional materials, these decisions typically will be made by a single trainer 
or a student. The basic aim of a rational (rather than promotional) dissem- 
ination effort will be to assist the user in acquiring needed information, 
bearing on the user*s own requirements, regarding all reasonable alternatives. 
Perhaps the aim will also be to assist the user in developing a frame of 
reference fDr evaluating the alternatives, and to make useful comparisons 
among them. To do this, one has to anticipate a user's requirements and of 
how the user processes information regarding alternatives. The applied research 
performed at the Far West Laboratory in developing the ALERT system suggests 
that v/ays must be found to help the user to quickly narrow a large set of 
alternatives down to a relatively small number (say five to seven), and to 
quickly review major characteristics regarding content, method, level of 

difficulty, rationale, cost, etc., to focus on two or three most promising 
O 
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alternatives. However, once this stage is reached, the user typically wants 
much more information than can usually be supplied in any one report. He 
also usually does not want much of the information which is often provided. 
Practical comments about pluses and minuses derived in actual applications, 
comparisons with popular, well-known alternatives, and practical critiques 
by users are usually more valued than highly technical field-test data, or 
analyses of characteristics that do not relate to the user's decision problems. 
The most valuable information is that which best helps the user to differen- 
tiate between alternatives (which may be of almost similar overall merit) in 
a way that permits the user to select the one with the greatest utility for 
his specific need. 

Evaluation .'procedures that attempt to maximize differences with respect 
to general assumptions about users' needs may in fact have little value 
in aiding the user in making such differentiations. This relates to an issue 
often ignored in evaluating alternative educational products: much more 
attention must be given to determining what information will best help users 
to make choices among good alternatives if a rational dissemination strategy 
is desired. Attention too often is focused on arriving at a good overall 
evaluation that at least correctly rank orders alternatives. This is suffi- 
cient only when gross selection, say for entry in a catalogue or for general 
promotion, is the aim. 

This implies a possibly important conclusion; namely, that user-oriented 
evaluation of instructional materials must be sequential and that different 
evaluation strategies are indicated at the different stages. At the first 
stage, for the purposes of general selection , evaluations based on overall 
merit assuming some general set of user needs seems to be the indicated 
strategy. At a second stage, for the purposes of classification (say in 
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identifying which materials would work best for specific applications), the 
amount and kind of evaluation information provided needs to be considered 
in terms of how it will provide the average user (or specified subclasses 
of iisers) with information having greater utility for cost than the user 
vmuld normally obtain. 

As one begins criticism or evaluation designed to assist users in specific 
decision situations , information concerning overall merit, or even character- 
istics that only generally differentiate among alternatives, will usually not 
be as useful as information relating to selecting the best instructional 
material for specific uses. In sum, greater attention must be given to 
differences in uses than to differences of the materials themselves.* For 
a general idea of what information users want, consult ^'Needed: Exportable 
Models of Significant Change in Education" (Ohme, 1972); although the article 
deals with educational practice models and is hardly definitive, it is useful. 
3 . Practi c al Recommendations 

Given the above discussion of issues, what practical recommendations can 
be made? Since specific recommendations can be given only when purposes, 
objectives* ^^esources and constraints are well-defined, we doubt there can be 
any one best evaluation procedure. 

In cases where the selection ratio is extreme, either in the sense of selec- 
ting or rejecting almost all, then a sequential (two-stage or multistage) selec- 
tion method should be employed. When the selection ratio is not extreme, for 

Ik 

Utelv, educational research has been concerned with aptitude-trea&nent 
interactions, based on the assumption that the same treatment may not he 
best for subjects with different aptUudes- If we consider instructional 
materials as "treatments" and translate "aptitude'' to mean the -i^tire 
set of significant user conditions, there is a clear parallel. What this 
aptitude-treatjnent interaction research also suggests is that aUhough the 
theory is good, obtaining useful empirical data to guide practical decisions 
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example when something like half of the candidates will be selected, then a 
one-stage, accept or reject method may be nearly as effective. In the case 
of extreme selection ratio cases, we believe that the methods developed by 
ETS (with the assistance of ERIE) are the most promising and well worked out 
(Epstein, Margosches, Schrader and Walton, 1971; Walton, et. al_. , 1973). 
The purpose of the ETS effort was to identify from a large product pool a 
relatively small group of educational products whose widespread dissemination 
and use would offer favorable odds for significant improvement in the quality 
of education. Basically, the approach consists of collection, classification, 
and organization of information on products, which is then reviewed by a 
panel of experts. This information is organized in a dossier that includes 
a carefully written precis of the information deemed most ijmportant to the 
paneVs evaluation needs. Because of the importance of conveying accurate 
and adequate information, these precis should be sent to the product deve- 
lopment principal investigators for their review and comment. The precis is 
the key document used in panel consideration although panelists could resort 
to the complete dossier. The panelists review the precis according to a set 
of criteria that have been cogently summarized on a single rating form. 
Major criteria are goals, evidence of effectiveness, costs, and adoptability. 
Panelists are permitted to arrive at their final overall rating in any way 
they chose. Special procedures are used to overcome well-known biasing 
effects resulting from the order in which products are examined. After 
initial independent ratings, the full panel meets to discuss and rerate 
products fur a semifinal rating. Products receiving high total ratings 
are listed on a blackboard in overall rank order based on the sum of 
individual overall ratings. A tentative cutting sccr is agreed upon, and 
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following further discussion of products in the vicinity of this score, some 
adjustments are made in ratings of products near the threshold. At this 
point a final set of ratings and a firm decision regarding the cutting score 
is made. The 1971 ETS report notes th?.t the use of a rating scale formulated 
in terms of the action to be taken by the panel and the ur^e of a numerical 
method for combining ratings from different raters worked very well and appears 
to have facilitated the setting of a standard for acceptance (Epstein, et al . , 
1971, p. 53). 

The ETS method is. expensive. Where large numbers rf products ave involved, 
one or more stages of prescreening by possibly just a pair of competent judges 
might be used to reduce the number of candidates to manageable size for consi- 
deration by a panel. As noted previously, there is no reason why a similar 
procedure would not work well when the need is to reject a few candidates. Such 
a procedure might be appropriate where severe criticism could be anticipated re- 
garding the justification for elimination of some instructional materials. 

When selection ratios are not extreme, the ETS method is not recommended 
primarily because it is far too costly; and a much less expensive process is 
needed. Unfortunately, we know of no really practical methods. The ETS method, 
as well as other analytical methods, depend heavily on the quality of product 
information available. As investigators at ETS (Walton, ei.al-» 1973), AIR (Dunn, 
et al., 1973), and the Far West Laboratory (1972) have learned, obtaining this 
information can be most difficult, time-consuming and expensive. The "PARADE" 
form (Dunn, et al. , 1973) evaluates a small number of products developed initially 
under USOE, NCERD sponsorship, but hardly covers more than a small fraction of 
conceivably eligible instructional materials. The PARADE form is only an iden- 
tification and classification aid and is not adequate by itself, a fact acknow- 
ledged by its senior author (personal comiiiunication with >. James Dunn). 
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This situation seems to leave the choice of evaluation method between 
depending on making judgments about the product itself or depending on the 
product plus whatever information can be obtained from developers, distributors 
or users. If the product alone is used, several analytical schemes at least 
suggest how one might proceed. The models by Easley, Jenkins, and Ashenfelter 
(1967), Stevens and Morrisset (1968), Hash (1969, 1970) and Walberg et al, 
(1962) are described and evaluated in the evaluation report. We concluded 
that, of the various content analysis approaches, Eash's seems to be the most 
promising for accomplishing rough screening and perhaps classification of 
instructional materials. Its chief virtues are its focus on estimated effec- 
tiveness rather than simple analysis and its deliberate intent to serve as 
a practical instrument of manageable length and complexity. 

The Eash assessment instrument can be employed when only the instruc- 
tional material is in hand. The Stevens and Morrissett Curriculum Analysis 
System (and its Far West Laboratory derivative) usually require additional 
information. The Eash assessment instrument requires much les^ time to 
complete than the Curriculum Analysis System, which, in turn, is usually less 
time consuming than the Easley, Jenkins, and Ashenfelter scheme for analysis 
of many assignable units within an instructional package. The Eash assessment 
instrument does deal with the goals and effectiveness (and overall rating) 
rubrics of the ETS rating form. It does not deal directly with the costs and 
adoptability rubrics. These two elements should, and probably can be, 
easily added. 

We believe that a selection instrument derived from the Eash assessment 
instrument but modified to consider "costs" and "adoptability" would be useful 
as an intermediate screening device when a large number of materials are to be 
considered and when the selection ratio is not extreme. Prior screening with 
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a less expensive device seens warranted. 

A three-stage process would probably be most economical. The first 
stage would be collectir opies of all candidate instructional materials, 
grouping them by categories, and then preliminary screening of candidates 
within each category to eliminate obviously obsolete, erroneous, poor, weak, 
stereotyped, or redundant items. For this purpose, the Easley, Jenkins, 
and Ashenfelter criteria are recommended. 

At the isecond stage, an "Eash-like" assessment instrument, modified 
to include cost and adoptability criteria, should be employed for classifying 
surviving instructional materials into three classes: (a) accepted, 
(b) rejected and (c) held for further evaluation.* At least two trained 
judges should independently assess each product. When they cannot agree on 
the classification of a product, it should be placed in the hold category. 
A second round of assessment, this time employing a new set of judges, should 
then be undertaken to reclassify all products placed 1n the hold category; 
however, all three classifications would still be allowed at this stage. 

At the final stage, a third set of judges would be employed. Their 
first task would be to make a blind reassessment exclusively of random 
samples of accepted and rejected instructional materials. The judges would 
be required to accept or reject all items. This step is desirable as a 
check on the inter judge reliability of the screening process. If satisfac- 
tory agreement is not reached, the entire effort would have to be repeated, 
presumably after further training of judges or reworking of the procedures 
or both. The final step for the third phase would be to dispose of the 

*If the assessment process entails considerable cost or risk, further 
development and pilot testing of Instruments and procedures is strongly 
Indicated. 
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remaining materials still in the hold category by accepting or rejecting them. 

The assessment protocols developed by the Eash selection instrument can be 
used as a basis for preparing annotations on all accepted entries. As with 
the ETS pre^cis, we recommended that developers be permitted to review and 
comment on these annotations before they are edited and finally published. 
(See Walton, et a^. , 1973 ). 

Summary . In the preceding section we have recommended two different 
methods of evaluation, depending on whether the intent is to select (or 
reject) only a few of a large set of candidate materials or to select some 
intermediate proportion (e.g., between 20 percent and 80 percent). The focus 
has been primarily upon selection; however, both recommended procedures create 
documentation that can be used as a basis for "classification" or "criticism." 
If more extensive analysis is desired, the Curriculum Analysis System is 
recommended. We prefer the Far West Laboratory version which has been sub- 
jected to several years of development and field test. These procedures have 
been documented in three items (unpublished, but obtainable from the Far 
West Laboratory): (a) Curriculum Analysis Team Instructional Manual, (b) The 
ALERT Mini-Notebook: A Guide for Gathering Information for Cards and Summaries, 
and (c) ALERT Information System Writer's Manual. 

Unfortunately, none of the recommended procedures (whether ETS's for 
extreme selection ratios or the Curriculum Analysis System for intensive 
criticism and evaluation) provides a substitute for experienced judges who 
will usually have to make evaluations with little or no field test data. 
Adequate field test data is simply not available for more than a very small 
portion of Instructional materials in educational RDD&E. Hence the rigorous 
evaluations based on such data (Stake, 1967; Epstein, et al. , 1971; Walton, 
et. al., 1973; Scriven, 1973) cannot be applied uniformly and, in fact, will 
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be applicable only in relatively rare instances. In most cases, performance 
effectiveness will have to be inferred. If this must be done, then Hash has 
at least given us a start by stating relevant criteria. 

A completely feasible and not too expensive next step would be to compile 
a set of instructional matf^rials for which there is reasonably aidequate field 
test evidence, and then to determine whetlier judges (who are unacquainted with 
the field test results) can make valid performance inv'erences based only on 
their examination of the instructional materials. If we must depend primarily 
on judgment , we could at least attempt to validate the judgment process. 

In closing, we must emphasize that there is no one set of criteria that 
can be applied optimally to all instructional materials evaluation situations. 

Ideally, one searches for relevant, cost-effective, attractive usable 

materials. Credible performance-effectiveness data based on comparable 
populations and comparable use conditions is highly desirable. In reality 

the information pertaining to such ideal criteria is quite limited and imper- 
fect. To paraphase a quotation, the assessment of educational (RDD&E) instruc- 
tional materials is the art of arriving at sufficient conclusions from insuf- 
ficient data. The basis of this art is not a list of criteria, nor even 
perhaps a set of rating scales. Rather it is in the development of an ex- 
plicit and sufficient statement of purpose and objectives; in defining as 
precisely as possible the decision situation(s) which classes of users may 
encounter vis a vis defined sets of instructional materials; and then in 
relating available (or obtainable) information about instructional materials 
to anticipated outcomes in these situations. This is an extremely complex 
process--so much so that some expert judges who attempt it, find it difficult 
to explain. It is primarily an art. In the absence of systematic efforts 
to analyze and validate the process, it will remain an ^rt. Procedural 
O Is, dossiers, precis, checklists, rating forms, and the like may make 
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the art more reproducible and possibly more efficient. But effectiveness 
criteria call for a massive inferential leap, from whatever is known or can 
be observed of the instructional material, to some set or sets of applications, 
and costs, and user's values (utilities) for those outcomes and costs. 

We have eschewed defining a particular evaluation process. But we have 
analyzed the general evaluation problem and made recommendations on evaluation 
; procedure. What we most stronilv urge is that any selected process be treated 
as itself a development task, which should be subjected to test, evaluation, 
and revision . If at all possible, the test data should attempt to assess, (no 
matter how crudely) product utility for target audiences. Until this effort 
is made, our evaluations of instructional materials will remain primarily 
guesswork. If it must be guesswork, NIE is advised to use judges who have 
extensive relevant experience in the actual use of instructional materials, 
preferably with populations of subjects, conditions, and content similar to 
those pertinent to RDD&E training. 
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VII SUMMARY 

A. Graphic Surmary 
The data presented in Chapters II, III, IV and V can be organized in terms 
of the following questions: 

1. Which users should be served? 

2. What are the purposes for which users seek infonnation? 

3. What information content do users want? 

4. What quantity of RDD&E literature is relevant to their information 
content needs? 

5. What quantity of instructional materials is available? 

6. Which proposed products and services are most valued? 

The figures on the following pages provide graphic summaries of key data 
relating to each of the above questions. The areas in each figure are proportional 
to overall weighted averages of priority points reported in data tables in Chapters 
II, III, and IV or percentages of counts reported in tables in Chapter V. 
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1. Which users should be served? 
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2. What are the purposes for which users seek 
information? 



FIGURE 2 
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3. What information content do users want? 
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4. What quantity of RDD&E literature is available relevant to 
their information content needs? 



FIGURE 4 

ESTIMATED RELATIVE QUANTITIES OF LITERATURE BY 
CONTENT CATEGORIES BASED ON ERIC COUNTS 
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5. What quantity of instructional materials is 
available? 



FIGURE 5 



ESTIMATED RELATIVE QUANTITIES OF BOOKS AND OTHER 
INSTRUCTIONAL MATERIALS BY CONTENT CATEGORIES 
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6. Which proposed products and services are most 
valued? 

FIGURE 6 

RELATIVE PREFERENCES FOR ALTERNATIVE 
INFORMATION PRODUCTS AND SERVICES 
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B, Conclusion 

The major challenge for this project was to study and to identify feasible, 
alternative solutions to user information requirements. Although the focus of 
the project was on "personnel and training," we believe that our findings have 
broader implications for the entire field of educational RDD&E. 

Our study started with certain assumptions derived from research into the 
behavior and information-using habits of scientists and engineers, from several 
years of experience in the development and field evaluation of educational 
R&D information systems, and from personal experience with the problems of 
obtaining information regarding educational RDD&E personnel supply and demand, 
training programs, and materials and resources* This experience had prepared 
us to anticipate a poorly organized and highly inefficient knowledge transfer 
ne twork . 

The studies of scientific information exchange in education by Nelson, 
Garvey, and Lin (1970) and Nelson (1972) have confirmed this impression. The 
educational "research" information network is indeed poorly structured and the 
typical educational researcher, as an information consumer, greatly lacks 
awareness of current activity. Our own study indicates that the problems for 
educational developers and diffusion personnel are even more serious. First, 
they are probably much less print-orientfsd than their discipline-oriented 
research colleagues. Secondly, the nature of their work typically requires 
reference to information in several fields. Finally, much of the information 
they seek (methods, techniques, procedures) is either not published at all, is 
given very limited distribution, or is not effectively indexed or easily retrieved 
from available information systems. The problems faced by federal sponsors appear 
^ to be much like those of the developers, except that the time pressures on the 
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sponsors are often greater and the needs for comprehensive, "decision-oriented" 
information is even more severe. 

Trainers were once in a more enviable position when only traditional 
research was the name of the game and the supply and demand situation was 
stable. But as demands increased for talent trained in DD&E and for personnel 
who could accomplish problem-solving disciplined inquiry in culturally and 
socially charged situations, training programs have responded slowly. Trainers 
who are aware of these changes report the need for, but difficulty in obtain- 
ing, relevant information and instructional resources. Employers are concerned 
to find persons trained to perform effectively in applied research, develop- 
ment, diffusion, evaluation, and R&D policy and management positions. They also 
require training resources to upgrade these, skills in their staffs. 

Most of these educational RDD&E personnel must choose between reliance 
on informal communication, which may not extend much beyond the work team or 
a highly "disorganized" invisible college, and reliance on formal communication 
channels and media, which deliver information years late and are heavily "research" 
biased in both content and the effectiveness of their retrieval of information. 

This rather dismal picture implies that RDD&E personnel in general, and 
more specifically those concerned with the R&D "infrastructure" (knowledge 
transfer, personnel capabilities, training capacities) may encouriter great 
difficulty in meeting their needs for information. 

We began with the belief that if the actual problems of transfer and 
utilization of information are to be solved, a complex of networks must be 
established and supported by appropriate services and hard-copy products that 
would guide potential target users in their inquiries and provide relevant 
information in acceptable, usable, inexpensive forms that will foster appro- 
priate and repeated use. Our field test experiences generally confirm the 

ERIC 



in 



validity of this belief. 

But we learned that the "research training'* audience is probably too 
small and certainly too "information apathetic" to justify the creation of a 
special information center solely dedicated to serving their needs. Yet it 
was clear that, with the possible exception of the researcher, nearly everyone— 
sponsors, performers, trainers, and employers— suffers formal structural and func- 
tional inability to generate, transfer, and utilize information and instructional 
resources more efficiently. The problem is not one that can be solved by 
information technology alone. We must look as well to the norms and values 
of our educational scientific and technical confmiunity, to the rewards and 
sanctions provided by sponsors, membership groups, and reference groups, and 
to the relevance of current educational RDD&E training and continuing education. 

We iraist understand that information transfer regarding personnel and 
training issues in educational RDD&E is "problem-oriented," that the larger 
areas of educational RDD&E itself are "field-oriented" and that the science 
which supports that effort is "discipline-oriented" (Hood, McCutchan, Aldrich 
and Wolf, 1972). Paisley (1971) notes that the source of the knowledge base 
of each "orientation" has different implications for the most effective 
organization of information functions and for the modes of communication. 
, Designs for educational RDD&E information systems have suffered from confusing the 
discipline-oriented information systems of the sciences with field-oriented infor- 
mation systems required by education. We suspect similar errors will be made in 
confusing problem-oriented information systems with field-oriinted information syst 

There seems to be little danger that NIE will make such mistakes soon , 
primarily because its staff are still sorting out what they have inherited from 
the Office of Education and from other agencies and are attempting to develop a 
more attractive set of R&D options and a more defensible rationale for future 
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R&D investments. Given the cutbacks in total federal R&D funding, a projected 
oversupply of scientific talent, severe retrenchment in federal support for 
scientific training, and valid reservations about the effectiveness of at least 
some educational R&D training strategies, there may be no immediate, compelling 
need to support the specific requirements of the userc>that have been documented 
in this report. The recommendations made in Chapter III regarding promising dissem- 
ination alternatives are all quite modest. This was deliberate, because we did not 
believe that more costly efforts could be justified, given the present situation. 
There is a valid need, but not a strong demand. The later must be cultivated. 

It would be a grave mistake, however, not to recognize the need and the 
opportunity for improvement in the communication of information, with regard not 
only to the relatively exotic area of "RDD&E personnel and training," but 
many problem-oriented areas which NIE now addresses. This study has not found 
a solutions but hopefully it may help to define the problem and to point to some 
possible first steps. 
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APPENDIX A 
LIST OF PROJECT REPORTS AND PRODUCTS 

Reports 

Design of a Personnel and Training Information System for Educational 
R&D Personnel : 

1. - Report of the Planning Phase , Paul D. Hood, Nancy A. McCutchan, 
Charles L. Aldrich, and William C. Wolf, Jr., September 1972. 

2. - Report of the Preliminary Design Phase , Paul D. Hood and Nancy A. 
McCutchan, December 1972. 

3. - The Domain of R&D Training Resources , Paul D. Hood, August 1973. 

4. - Final Report , Paul D. Hood and Nancy A. McCutchan, August 1973. 

5. - Evaluation of Instructional Materials for Educational RDD&E Content 
Areas, Paul D. Hood, July 1973. 

Project Products 

Design of a Personnel and Training Information System for Educational 
R&D Personnel : 

1. - Educational RDO&E Personnel and Training Abstracts > Nancy A. McCitchan, 
Carol Burkhart, and Paul D. Hood, August 1973. 

2. - Recommended Journals and References , Frank W. Mattas, August 1973. 

3. - Recommended Books and Monographs , Paul D. Hood, August 1973. 

4. - Personnel and Training Newsletter , Nancy A. McCutchan, ed. Three 
issues: vol. 1, no. 1, November 1972; no. 2, December 1972; no. 3, April 1973. 

5. - Personnel and Training Abstracts , Nancy A. McCutchan, ed., no. 1, 
November 1972. 

6. - Catalogue of Educational RDD&E Instructional Materials , February 1973. 

7. - A Source Book of Educational RDD&E Literature (Sample), N.D. 

8. - A Source Book of Educational RDD&E Case Studies (Sample), N.D. 
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Technical Papers and Draft Reports 

1. Creation of "Cost-Preference" Materials . March 1973. 

2. User Study . March 1973 

3. The Domain of R&D Training Resources . April 1973. 

4. Promising Alternatives for the Dissemination of Educational RDD^E Personnel 
and Training Information , April 1973. 
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